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the reactions of the participants and their supervisors toward the 
effectiveness of the training programs. Phase 2 of the study consisted of 
field testing the instrument in two local community college districts. Based 
on the data collected from the study, it was concluded that the programmatic 
effectiveness of workforce training programs could be evaluated immediately 
using the instrument developed for this study; however, the personal 
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ABSTRACT 



THE DEVELOPMENT OF AN INSTRUMENT FOR ASSESSING THE REACTION 
OF PARTICIPANTS AND THEIR SUPERVISORS TO THE EFFECTIVENESS OF 
WORKFORCE TRAINING PROGRAMS IN THE MISSISSIPPI DELTA 



by Marjorie Taylor 
EDD, Delta State University 
Cleveland, Mississippi 
December, 2000 

Major Committee Chair: Jane L.Weare 



In 1 994, the Mississippi legislature enacted the Workforce Education Act, Senate 
Bill 2955, which called for the creation of One-Stop Career Centers affiliated with each 
community or junior college district. The mission of the One-Stop Career Centers was to 
provide assessment, training, and placement services to individuals needing retraining, 
training, and upgrading for local industry. 

At the time of the present study, the only standardized form of collecting 
programmatic data from the One-Stop Career Centers to determine their effectiveness 
was the Legislative Accountability Report. The purpose of this study was to develop an 
evaluation instrument to assess the effectiveness of workforce training programs provided 
by Mississippi’s One-Stop Career Centers. The evaluation instrument was designed to 
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assess the reaction of participants and their supervisors toward the effectiveness of 
training programs conducted by the One-Stop Career Centers. Nancy D. Padak and Gary 
M. Padak’s workplace training effectiveness categories of programmatic, personal, and 
external provided the framework for this instrument. 

The study was conducted in two phases. Phase I was the development and pilot 
test of the instrument for recording the reactions of the participants and their supervisors 
toward the effectiveness of the training programs. Phase II of the study consisted of field 
testing the instrument in two local community college districts. Phase II results were 
used to compare the responses of the training program participants and their supervisors. 

Based on data collected from the study, it was concluded that the programmatic 
effectiveness of workforce training programs could be evaluated immediately using the 
instrument developed for this study. However, the personal effectiveness and the external 
effectiveness of training programs could not be evaluated immediately using the 
instrument developed for this study. 

From the findings of the study, the researcher recommended that it be continued 
with several workforce training programs. The evaluation instrument should be 
administered to the supervisors and participants after they have had ample time to use the 
skills learned and observe changes in behavior. Additional research was needed in order 
to determine if there were differences in responses from participants in voluntary and 
mandatory training programs. 
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CHAPTER I 



INTRODUCTION 

In an effort to reorganize workforce preparation in Mississippi, the Mississippi 
Community College Foundation, a publicly funded not-for-profit corporation of 
community and junior colleges in Mississippi, formed the Millennium Group in 1992. In 
the Millennium Group’s report to the people of Mississippi, the group called for a “New 
Wave of Reform” in education (Mississippi Community College Foundation, 1993), one 
that included the education of adults. The future economic and social prosperity of 
Mississippi depended on a well-educated workforce. 

The Millennium Group recognized the immediate need for an institutionalized 
statewide delivery system of workforce training. The Group recommended that 
Mississippi’s community colleges serve as the primary, if not sole, providers of 
workforce training, thus transforming them into “major economic and workforce 
development tools” (Mississippi Community College Foundation, 1993, p. 10). The 
Millennium Group cited an immediate recommendation to “establish one-stop career 
centers on each campus with the capacity to recruit, assess, counsel, and place in jobs or 
training anyone who is or ought to be in the labor force” (Mississippi Community College 
Foundation, 1993, p. 10). 
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Based on the immediate recommendations of the Millennium Group, 

Mississippi’s Workforce Education Act of 1994, Senate Bill 2955, was enacted by the 

Mississippi legislature. This legislation called for the creation of One-Stop Career 

Centers affiliated with each community or junior college district (S. 2955 § 37-153-1 1, 

1994). The mission of the Workforce Education Act of 1994 was as follows: 

... to provide for the creation and development of a regionally based system in 
Mississippi for education and training which: responds to the needs of 
Mississippi’s workforce and employers; is driven by the demands of industry and 
a competitive economy; makes maximum use of limited resources; and provides 
for continuing improvement through constant assessment of the results of 
education for individual workers and employers. (S. 2955,1994) 

It has been six years since the enactment of Mississippi’s Workforce Education 

Act of 1994. Has the mission of the legislation being achieved? How are the One-Stop 

Career Centers functioning in their individual quest to fulfill the mission? And what 

adjustments, if any, need to be made to better meet the demands of industry and the 

economy? These questions provided the foundation for the present study. 

The most effective and reliable way to answer these questions was to analyze 

evaluative data from each One-Stop Career Center. At the time of the present study, the 

only standardized form of collecting programmatic data was the Legislative 

Accountability Report (see Appendix A). One-Stop Career Centers sent this report 

annually to each organization for which they had provided training. Data were compiled 

and reported to the State Workforce Development Council and the State Legislature. Any 

other evaluative data which may have been collected indicating the effectiveness of each 
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One- Stop Career Center’s training programs remained at the discretion of the individual 
Center director. 

One-Stop Career Center directors at Mississippi Delta Community College and 
Coahoma Community College indicated that the Legislative Accountability Report did 
not adequately assess the effectiveness of their individual One-Stop Career Center’s 
programs in meeting the needs of the employers in their districts (personal 
communication, October 12, 1 999). They both realized the necessity for periodic 
program evaluations to ensure continuous improvement. 

Statement of the Problem 

The problem of this study was to establish a method of collecting evaluative data 
indicating the effectiveness of workforce training programs conducted by Mississippi’s 
One-Stop Career Centers. 



Purpose of the Study 

The purpose of this study was to develop an evaluation instrument to assess the 
effectiveness of workforce training programs. Nancy D. Padak and Gary M. Padak’s 
workplace training effectiveness categories provided the framework for this instrument. 
Padak and Padak’s (1991) effectiveness categories included programmatic, personal, and 



external factors. 
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Programmatic factors included the program structure and the program content. 

The personal category included measures of effectiveness directly related to adult 
learning, such as academic achievement and quality of life. The external category 
included context-based and financial factors. Context-based factors referred to the rate of 
participation and the extent to which the program met the needs of identified external 
groups. Financial factors included the extent to which the participant’s earning power 
had increased as a result of the training. 

The evaluation process was multi-level, consisting of the participation of the 
employees and their supervisors. The employees and their supervisors responded to the 
questions on the instrument. A comparison of the descriptive data was made. 

Conclusions were drawn as to the effectiveness of the workforce training program. 

The present study was conducted in two phases. Phase I was concerned with the 
development of the instrument for recording the reactions of the participants and their 
supervisors toward the effectiveness of the completed workforce training programs. The 
instrument was designed to be completed quickly and easily to ensure the participation of 
the management staff (McMillan & Schumacher, 1997). 

The instrument was piloted in one community college district. The pilot test for 
this instrument was conducted for the purpose of analyzing each item. Results from the 
pilot test were used by the researcher to make necessary revisions to the items. 
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Research Questions 

The study proposed to answer the following question in Phase I: 

1 . What items should be included on the instrument to effectively assess both 
participant and supervisor reactions in the three effectiveness categories 
(programmatic, personal and external)? 

Phase II of the study consisted of field testing the instrument in the districts of 
Coahoma Community College and Mississippi Delta Community College. Phase II 
results were used to compare the responses of the participants and their supervisors. 

The study proposed to answer the following questions in Phase II: 

1 . Is there a difference between the reactions of participants and the reactions 
of the participants’ supervisors to the programmatic effectiveness of the 
recently completed workforce training program? 

2. Is there a difference between the reactions of participants and the reactions 
of the participants’ supervisors to the personal effectiveness of the recently 
completed workforce training program? 

3. Is there a difference between the reactions of participants and the reactions 
of the participants’ supervisors to the external effectiveness of the recently 
completed workforce training program? 
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Delimitations 

The following delimitations applied to the study: 

1. The study was conducted with organizations which, had participated 

in or, were participating in training programs with two of Mississippi’s 15 
community colleges. 

2. The instrument developed for this study was the only instrument used 
to assess the reaction of participants and their supervisors to the 
effectiveness of the workforce training programs. 

Assumptions 

The following assumptions were made regarding this study: 

1 . The organizations that participated in this study were representative of the 
organizations receiving training through Mississippi’s One-Stop Career 
Centers. 

2. The two community colleges that participated in the study were 
representative of other community colleges in Mississippi and in the 
nation. 

3. Data gathered from the administration of the evaluation instrument were 
assumed to be true responses from employees and their supervisors. 
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Definitions 

Interpretation of the study required the definition of the following terms: 

Basic skills specialist . The One-Stop Career Center staff member responsible for 
the design, implementation, and coordination of workforce training programs focusing on 
reading, writing, and mathematical skills. 

Industrial coordinator . The One-Stop Career Center staff member responsible for 
the design, implementation, and coordination of workforce training programs focusing on 
the technical aspects of the business; such as, machine operation skills, safety skills, and 
quality control techniques. 

Legislative Accountability Report (LARI . The method for providing 
accountability and quality assurance for the One-Stop Career Center system within 
Mississippi’s State Board for Community and Junior Colleges. 

One-Stop Career Centers . The affiliate of the community college responsible for 
providing assessment, training, and placement services to individuals needing retraining, 
training, and upgrading for local industry (S. 2955 § 37-153-11, 1994). 

Program evaluation . Planned process of gathering and analyzing data to help 
make better decisions concerning a program. 

Reaction . The level of evaluation indicating how participants feel about the 
training program (Kirkpatrick, 1998). 
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State Workforce Development Council . Created through Mississippi’s Workforce 
Education Act of 1994 to serve in an advisory capacity for the One-Stop Career Centers 
and monitor their effectiveness (S. 2955 § 37-151-69, 1994). 

Su pervisor . Person in direct charge of the employee. 

Trainee . Participant in an education program. 

Training program . A synonym for workforce training program. 

Training program effectiveness . The extent to which the training program meets 
the intended or expected results. 

Workforce . All people working or available to work. 

Workforce specialist . The One-Stop Career Center staff member responsible for 
the design, implementation, and coordination of workforce training programs focusing on 
application-oriented educational tasks; such as, teaching job-specific reading 
competencies with the use of actual job-related reading materials. 

Workforce training program . Any educational initiative based on the objectives 
of the business that is designed to improve one’s skill requirements of the job. 

Design of the Study 

This study was conducted as a developmental study, the result of which was an 
evaluation instrument used in assessing the effectiveness of workforce training programs. 
The evaluation instrument was developed with the assistance of workforce training 
experts in Mississippi’s One-Stop Career Centers. The instrument was piloted in one of 
Mississippi’s community college districts. 
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Field testing of the instrument occurred in two of Mississippi’s community 
college districts. Results were used to compare the responses of the participants with 
those of their supervisors. The instrument served as a model which could be 
implemented statewide. 

Summary 

The preceding material presented in this chapter outlined the purpose of the study. 
Chapter II will address the review of literature as it pertains to the present study. Chapter 
III will present procedures used in this study. Chapter IV will present the data collected. 
Chapter V will present the conclusions drawn from this study. 
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CHAPTER II 



REVIEW OF RELATED LITERATURE 

The purpose of this study was to develop an evaluation instrument to assess the 
effectiveness of workforce training programs. Chapter II reviews the literature of 
research in areas related to the present study. The information is subdivided by topic. 
First to be considered is research focused on the need for workforce training, both in the 
South and in Mississippi. Secondly, research that describes training program evaluation 
will be reported. Lastly, research dealing with developing and administering 
questionnaires will be presented. 

Need for Workforce Training 



The South 

The nation, particularly the South, is facing an impending workforce crisis. 
Globalization and technological innovations are rapidly changing the nature of the 
workplace (Judy & D’Amico, 1997). These changes profoundly affect the American 
worker. The South is experiencing rapid growth in high-skill, high-wage occupations, 
while low-wage, low-skill occupations are growing more slowly. Judy & D’Amico 
further state that unless the education and skill levels of the American workforce are 

10 
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upgraded, America’s productivity and prosperity will suffer. Investing in education and 
training over an entire lifetime has now become a fact of life. Lifelong learning has 
become the norm rather than the exception (National Alliance of Business, 1998). 

The South’s economy was fueled during the 1970's and 1 980's by a stream of 
educated young people entering the workforce (Manpower Development Corporation 
[MDC], Inc., 1996). The elimination of legal racial segregation in employment and 
improvements in the quality of life made the South more attractive to business and 
industry. Today, however, there is a slow growth in the workforce. These demographic 
trends now present the South with a workforce imbalance (MDC, Inc., 1997). 

A report by MDC, Inc. (1996), The State of the South, acknowledges that for the next 15 
years, the South will have to compete in the global marketplace largely with the people 
already in its labor force. MDC, Inc. (1992, August) cites that the vast majority of 
workers in the year 2010 are already on the job today. Between 1996 and 2010, the 
region will have eight million more people between the ages of 45 and 64, while the 
number of workers between the ages of 20 and 44 will decrease by 800,000 (Judy & 
D’Amico, 1997). 

The eight million person “bubble” will amount to a substantial aging of the 
workforce (MDC, Inc., 1996, p. 3). Most of these people are already deep in their 
working lives and view formal schooling as a long-ago experience (MDC, Inc., 1996). At 
a time when the American workplace is demanding literacy, numeracy, and critical 
thinking skills, the current workforce is limited by educational deficiencies (MDC, Inc., 
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1992, September). The Southern economy remains heavily dependent on the skills of the 
“baby boomers,” most of whom have not retired and will still hold jobs in the year 2015. 

As a result, the region must provide massive education and retraining for the older 
adult workforce which will carry a disproportionate economic burden of supporting three 
million more elderly and almost one million more schoolchildren by the year 2010 
(MDC, Inc., 1997). New entrants to the labor force must be dramatically better prepared. 
This requires a new social contract on education and training for the region, one that 
broadens education reform to include education and training for adults (MDC, Inc., 1992, 
September). America, and especially the South, cannot sustain even its current level of 
prosperity unless it improves workforce skills. 

Community College System for Workforce Training 

The comprehensive community college system has become recognized as the 
leading provider of the workforce development needs of adults. MDC, Inc. (1998, 
September) reports that the South’s community colleges are the “workforce’s repair 
shop” as they prepare young people, uneducated adults, and college graduates with up-to- 
date skills. When they work in tandem with employers, the workforce training they 
provide can be a powerful catalyst for economic development (MDC, Inc., 1998, May). 

Zeiss and Associates (1997) note that community colleges have many strengths 
that put them at the center of an effective workforce development system. 
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These strengths include the following: 

1 . They provide flexible course offerings that fit the needs of businesses and 
students. 

2. They are in touch with the needs of employers for workforce training. 

3. They are equipped to educate nontraditional students. 

4. They are located within financial and geographic reach of most of the 
population. 

The knowledgeable worker is absolutely necessary for America to compete in a 
global economy. On average, employers report that one out of every five workers do not 
have the necessary skills for full proficiency on the job (Gore, 1998, January 12). 
Furthermore, 88 percent of companies have difficulty finding qualified applicants for at 
least one job function. The estimated total cost, including lost productivity, substandard 
performance, and unemployment insurance, of undereducated workers in the Southeast is 
in excess of $57.2 billion per year (Mendel, 1992). The key to America’s economic 
future is to better educate students and train existing workers. Clearly, the role of the 
community colleges in providing quality workforce training has never been more 
important (Zeiss & Associates, 1997). 

Mississippi’s One-Stop Career Centers 

Mississippi recognized the necessity of a dramatically improved workforce in the 
early 1990's. Workforce training was the focus of two major studies in the 1991-1993 
period. In 1991, the Commission on Work Force Excellence, a group of Mississippi 
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business leaders and state officials, completed a study on the effectiveness of federal and 
state funded training programs offered by four state agencies (Howell, 1996). Howell 
further stated that the commission recommended that an employer-driven Quality Work 
Force Council be established at the state level to coordinate development standards for 
workforce training programs. 

The second study entitled Building a New Workforce for a New Century was 
conducted in 1 993 by the Millennium Group. The Millennium Group was organized in 
late 1 992, following a study tour of the German dual educational system by two 
community college presidents and the Executive Director of the Mississippi Community 
College Foundation. Sixty-nine Mississippi leaders from every facet of business, 
industry, education, and public service agreed to serve on the task force (Mississippi 
Community College Foundation [MCCF], 1993). The charge of the Millennium Group 
task force was to research and develop a plan by which Mississippi’s workforce could 
become internationally competitive. 

The Millennium Group’s recommendations included the identification of 
Mississippi’s community and junior colleges as the one-stop sites that could provide 
workforce training to business and industry while offering counseling, assessment, and 
training services to those seeking to enter the workforce (Howell, 1996). This 
recommendation prompted a workforce training bill which was introduced in the January 
1994 legislative session. Howell (1996) cites that this piece of legislation worked its way 
through all committees and both houses to the governor’s desk without a dissenting vote: 
“an unprecedented show of support for a major piece of legislation” (p.552). The 
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workforce training system established by the Mississippi Workforce Education Act of 
1994, Senate Bill 2955, has since become a model for the nation (Vickers, 1995). 

One-Stop Career Centers are now located at each of Mississippi’s 15 community 
and junior colleges to deliver workforce training and services to industries and 
individuals. An employer-driven state workforce development council and 15 district 
councils are charged with setting the workforce training agenda for Mississippi (MCCF, 
1994). 



Training Program Evaluation 
Lack of Training Program Evaluation 

Organizations in America are spending in excess of $40 billion each year on 
training (Shapiro, 1995). Mississippi businesses contributed more than $21 million to 
workforce training in fiscal year 1 999 (Mississippi State Board for Community and 
Junior Colleges, 1999). Are these companies getting the quality training they deserve? 
Are they getting the maximum return on their investment? A well-designed, systematic 
program evaluation can help provide answers to these questions. 

Unfortunately, the effort devoted to training program evaluation and the state of 
evaluation methodology leave much to be desired (Goldstein, 1974; Bell & Kerr, 1998). 
Compared to the number of training programs, very few program evaluations have been 
conducted and published. Fewer than half of America’s training programs are formally 
evaluated (Camevale & Schulz, 1998). Tracey (1968) cites the lack of interest in 
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evaluation as a consequence of two factors: (a) the nature of training and development 
programs themselves, and (b) the absence of a suitable conceptual framework for 
meaningful evaluation. Shapiro (1995) asserts that no universally accepted model of 
training evaluation exists, hence the difficulty in the evaluation process. 

Research indicates a variety of reasons for the lack of adequate evaluation. 
However, there is a consistency in these views which includes the following: 

1 . There is considerable difficulty in establishing acceptable criteria for 
program evaluation (Tracey, 1968; Goldstein, 1974; Woodington, 1980; McEvoy, 1998). 

2. There is a serious lack of personnel trained in the methodology of 
evaluation (Tracey, 1968; Goldstein, 1974; Woodington, 1980; Hilbert, Preskill, & Russ- 
Eft, 1997). 

3. The scope and strategies of evaluation are greatly influenced by the nature 
and type of organization (Goldstein, 1974; Woodington, 1980). 

4. The personnel responsible for training are not responsive to the need for 
evaluation or are fearful of the entire process (Tracey, 1968; Goldstein, 1974; 

Woodington, 1980; Mikulecky & Lloyd, 1993; Boulmetis & Dutwin, 2000). 

Need for Program Evaluation 

People do not agree on one definition of evaluation. Most will agree, however, 
that evaluation is a process of collecting and analyzing data (Boulmetis & Dutwin, 2000). 
Boulmetis and Dutwin further state that the process is guided by the reason for doing the 
evaluation. Hence, training program evaluation may be defined as the process of 
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collecting and analyzing data for the purpose of making certain decisions about the 
efficiency, effectiveness, or impact of the program (Camevale & Schulz, 1998; Merwin, 
1992; Kirkpatrick, 1998). 

Research points to three critically important reasons for an adequate evaluation 
program: (a) fiscal purposes, (b) justification of the program, and (c) program 
improvement (Tracey, 1968; Kirkpatrick, 1998; Boulmetis & Dutwin, 2000). According 
to Tracey (1968), several fundamental assumptions underlie the need for evaluating 
training programs. The assumptions include the following: 

1 . Any training program must be validated; the efficiency and effectiveness 
of programs must be objectively determined. 

2. Any training or development program can be improved. 

3. Improvement of any training or development program can be affected by 
objective and coordinated evaluation of every aspect of the operation, the application of 
creative thinking by all personnel, deliberate collection of data from all personnel, critical 
analysis of all data, and systematic development and tryout of policies and procedures 
needed to carry out plans. 

Effective Program Evaluation 

An effective evaluation of training programs must be conducted in such a way that 
it is consistent with the purposes, objectives, and goals of the training activity. Research 
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indicates a common set of principles which should guide all evaluative efforts. These 
include the following: 

1 . Evaluation must be conducted in terms of purposes; in other words, the 
purposes of the evaluation process must be clear to all involved (Tracey, 1968; Phillips, 
1998; Rossi, Freeman, & Lipsey, 1999). 

2. Evaluation must be collaborative and involve all stakeholders in the design 
and implementation (Tracey, 1968; Merwin, 1992; Mikulecky & Lloyd, 1993; Hilbert et 
al., 1997; Phillips, 1998; Boulmetis & Dutwin, 2000). 

3. Evaluation must be continuous (Tracey, 1968; Goldstein, 1974; Robinson 
& Robinson, 1989; Merwin* 1992; Mikulecky & Lloyd, 1993; Hilbert et al., 1997; 
Kirkpatrick, 1998; Phillips, 1998; Rossi et al., 1999; Boulmetis & Dutwin, 2000). 

4. Evaluation must be specific to the program (Tracey, 1 968; Goldstein, 

1974; Merwin, 1992; Phillips, 1998; Rossi et al., 1999). 

5. Evaluation must include procedures for implementing and communicating 
the evaluation process and progress to all stakeholders (Tracey, 1968; Goldstein, 1974; 
Merwin, 1992; Mikulecky & Lloyd, 1993; Hilbert et al., 1997; Kirkpatrick, 1998; 

Phillips, 1998; Rossi et al., 1999; Boulmetis & Dutwin, 2000). 

According to Shapiro (1995), the measure of a training program’s quality is its 
effectiveness. Training is effective to the degree that it produces the desired behavior in 
the population being trained. Moreover, a training program is considered to be effective 
when each participant is able to use the knowledge and skills taught in the program to 
bring about a desired result on the job (Merwin, 1992; Garavaglia, 1998; Shapiro, 1995; 
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Boulmetis & Dutwin, 2000). Parry (1997) states that a training program is most 
successful when “the right participants receive the right knowledge, attitudes, and skills 
taught by means of the right methods, media, and instructor at the right time and place so 
as to meet or exceed the organization’s expectations” (p. 1). As reported by Camevale 
and Schulz (1998), the purpose of training is to improve performance, and the purpose of 
evaluation is to improve training’s effectiveness and efficiency. 

The key to effective training and evaluation is a needs assessment of the learner 
and the work environment before, during, and after training (Shapiro, 1995; Gordon, 

1996; Robinson, D. G., 1996; Parry, 1997; Rossi et al., 1999). This is an important 
concept in the “Training-for-Impact” approach as defined by Dana G. Robinson and 
James C. Robinson (1989). Unfortunately, many training and development professionals 
fall into the “Training-for- Activity” trap. According to Dana G. Robinson (1996), this 
mind set focuses strictly on the activity of training, i.e., the number and variety of 
programs conducted, the number of participants, and any other factors for which the 
training provider may be held accountable. 

In contrast, the “Training-for-Impact” approach establishes a “causal link” 
between a skill deficiency and a specific business problem (Gordon, 1996; Rossi et al., 
1999). This approach allows for much more effective training and evaluation. Dana G. 
Robinson and James C. Robinson (1989) define the focus of “Training-for-Impact” as 
being: 

results-oriented training that is driven by business needs, helps the organization 
achieve its goals, provides people with the skills and knowledge they need to 
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improve their performance, assesses readiness of the work environment to support 
learned skills, has management accepting the responsibility for a supportive work 
environment that encourages skills transfer, and has measurable results that can be 
tracked, (p. 10) 

These factors must be built into the design of the training system before any training takes 
place (Robinson & Robinson, 1 989). 

Program Effectiveness Categories 

The measures of success of a training program are the standards by which the 
value of the program can be judged. In developing categories of training program 
effectiveness, Nancy D. Padak and Gary M. Padak (1991) researched 19 program 
descriptions listing evaluative criteria. These criteria were grouped into three broad 
categories, each with subcategories, for determining program effectiveness: 

(a) programmatic, (b) personal, and (c) external. 

The category of programmatic includes information on the program structure and 
program content. Kirkpatrick (1998) lists examples of questions to ask in evaluating 
program structure and program content including: 

1 . Does the subject content of the training program meet the needs of those 



attending? 



2. Are the training facilities satisfactory? 



3. Is the schedule of the training program appropriate for the participants? 



4. Was the coordination of the training program satisfactory? 
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The personal category includes measures of effectiveness directly related to the 
participant. Subcategories consist of academic achievement and quality of life. 
According to Kirkpatrick (1998), academic achievement and quality of life can be 
evaluated by determining one or more of the following: 

1 . What knowledge was learned? 

2. What skills were developed or improved? 

3 . What attitudes were changed? 

4. To what extent did the participant’s self-esteem increase? 

External indicators of effectiveness include context-based and financial factors. 
Context-based factors refer to the extent the program meets the needs identified by 
external groups, such as the participant’s supervisor. The extent the program shows a 
return on investment is considered a financial factor. The return on investment may 
include a savings to the industry or an increase in the participant’s earning power 
(Kirkpatrick, 1998). 

Formative and Summative Evaluation 

Heideman (1998) notes that evaluation is both a systematic and a systemic part of 
all training development. Without systematic evaluation, there is no feedback to provide 
the information necessary to improve programs or quality information to make decisions 
(Goldstein, 1974). Research indicates two main types of evaluation which occur during 
training design: (a) formative, and (b) summative. 
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Formative evaluation is a systematic process in which assessments are made while 
development takes place. It occurs during the design and delivery of the training program 
(Goldstein, 1974; Robinson & Robinson, 1989; Isaac & Michael, 1995; Parry, 1997; 
Heideman, 1998; Boulmetis & Dutwin, 2000). Formative evaluation focuses on 
continuous improvements to the instructional materials and the instructional design 
process itself. Its purpose is to identify problem areas which can be addressed and 
modified while change is still possible and productive (Mikulecky & Lloyd, 1993). 
Information is collected to determine the appropriateness and effectiveness of the 
program’s learning activities and content (Robinson & Robinson, 1989). 

End-of-course evaluation is referred to as summative evaluation. This evaluation 
occurs after training has been completed and helps assess the worth of the training 
intervention (Isaac & Michael, 1995; Heideman, 1998). Parry (1997) refers to summative 
evaluation as the process of measuring the learner’s performance after training and the 
degree to which the behavior back on the job meets the expectations that prompted the 
training. According to Dana G. Robinson and James C. Robinson (1989), the primary 
purpose for summative evaluation is to identify the impact of the program on individuals 
and on the organization, thus measuring transfer of training and not simply acquisition. 
Summative evaluation is a way of determining how effective the program is. 

In order to evaluate training programs effectively, it is desirable to perform both 
formative and summative evaluations. Parry (1997) cites that evaluation must start before 
training begins, once again indicating the importance of a needs assessment to determine 





what will be evaluated. Conscientious training professionals should be concerned with 
results of both formative evaluation and summative evaluation. 
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Kirkpatrick’s Four-Level Evaluation Model 



During the past four decades, numerous models of training program evaluation 
have been proposed. No model has been as useful and widely used as the one developed 
by Donald Kirkpatrick at the University of Wisconsin in 1959 (Hilbert et al., 1997; 
Phillips, 1998). Originally called a “four-step approach” to evaluation, it has been called 
a model, a system, a framework, a taxonomy, a methodology, a typography, and a 
vocabulary. Kirkpatrick himself now calls it the four-level model of evaluation 
(Kirkpatrick, 1998). 

Kirkpatrick’s model was the first attempt to formalize the notion of training 
evaluation (Hilbert et al., 1997). Most models found in research literature either directly 
or indirectly build on Kirkpatrick’s model. Kirkpatrick noticed that training evaluation 
could be conducted with four possible training outcomes in mind: (a) reaction, 

(b) learning, (c) behavior, and (d) results. These four outcomes, or levels, represent a 
sequence of ways to evaluate programs. According to Kirkpatrick (1998), each level is 
important and has an impact on the next level. As one moves from one level to the next, 
the process becomes more difficult and time-consuming. 

Data is easiest to generate at the first level of the model and progressively harder 
to collect and to interpret as one moves up the model. The effects being measured are 
short range at Levels I and II and longer range at Levels III and IV. These ranges help 
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explain estimates that over 85 percent of all training programs are evaluated at Level I, 
while the number drops progressively with fewer than 10 percent being measured at Level 
IV (Parry, 1997). 

Kirkpatrick (1998) defines Level I as the measurement of learner reaction. 
Evaluation on this level measures how participants in the program feel about the training. 
This evaluation is commonly referred to as customer satisfaction. If training is going to 
be effective, it is important that participants react favorably to it. Participant reaction may 
be easily measured by the use of questionnaires or “smile” sheets. 

Level II of Kirkpatrick’s model includes evaluating learning. Learning determines 
the extent to which participants have improved or increased their knowledge or skills as a 
result of the training (Kirkpatrick, 1998). Increased knowledge is relatively easy to 
measure by means of a test related to the content of the program that is administered 
before and after the training. 

Level III consists of evaluating the behavior of the participants. The evaluation of 
behavior determines if participants are using or transferring their newly learned 
knowledge and behaviors back on the job (Hilbert et al., 1997). The process of 
evaluating behavior is complicated and often difficult to do. Behavior may be evaluated 
by the use of interviews with participants’ supervisors, survey questionnaires, 
performance simulations, or observation. It is important to allow sufficient time for 
behavioral changes to occur before evaluating. 

Level IV of Kirkpatrick’s model is defined as evaluating results (Kirkpatrick, 
1998). Evaluation at this level determines if the training has affected business results or 
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has contributed to the achievement of the organizational goals. Commonly referred to as 
retum-on-investment, this is by far the most difficult level to evaluate (Parry, 1998). 
Many factors make it difficult to measure the dollar value of the benefits of training and 
the cost of the training itself. Retum-on-investment may be calculated through a cost- 
benefit analysis or pre-training and post-training comparison of data already in the 
system, such as accident rate, new accounts, rejects, absenteeism, or turnover (Parry, 
1997). Level IV also requires ample time following training before the retum-on- 
investment can be calculated. 

Kirkpatrick’s Level I: Reaction 

Kirkpatrick’s first level of training program evaluation is the reaction or critique 
evaluation. Evaluation on this level measures how those who participate in the training 
program react to it. Kirkpatrick (1998) calls this a measure of customer satisfaction. 
Evaluation at Level I determines the participants’ degree of satisfaction with a program’s 
design and delivery. Participants’ satisfaction is a crucial piece of information in 
determining the effectiveness of the training program. If training is going to be effective, 
it is important that trainees react favorably to it. If participants are not satisfied with the 
learning experience, they probably will not use what they have learned and will 
undoubtedly advise others not to attend (Robinson, 1989). Kirkpatrick (1998) states that 
“positive reaction may not ensure learning, but negative reaction almost certainly reduces 
the possibility of its occurring” (p. 20). 
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Kirkpatrick (1998) cites four reasons why measuring participant reaction is 
important. These reasons are as follows: 

1 . The measurement of participant reaction gives valuable feedback for 
evaluating the training program as well as comments and suggestions for improving 
future programs. 

2. The measurement of participant reaction tells participants that the trainers 
are there to help them do their job better and that they need feedback to determine how 
effective they are. This gives the participants ownership in determining the effectiveness 
of the training program. 

3. Reaction sheets can provide quantitative information which can be shared 
with managers and others concerned about the program. 

4. Reaction sheets can provide trainers with quantitative information that can 
be used to establish standards of performance for future programs. 

Level I evaluation is considered to be both formative and summative (Robinson & 
Robinson 1989; Kirkpatrick, 1998). It is a summative evaluation in that it typically 
comes at the end of the program. Level I evaluation can be used to make improvements 
in the program, hence it is also considered to be a type of formative evaluation. 

Evaluating reaction is not only important but easy to do. Many training 
professionals belittle reaction evaluation saying that it produces no useful information. 
According to Dana G. Robinson and James C. Robinson (1989), this could be due to two 
problems causing dissatisfaction: (a) the purposes of the reaction evaluation may not be 
clearly identified or may be incorrect, and (b) the questions may be poorly constructed 
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and so may provide poor information. Robinson and Robinson also cite common 
mistakes made in constructing reaction evaluations. These mistakes include: 

1 . Creating one reaction evaluation to be used for every training program, 
reflecting the “one size fits all” idea, 

2. Creating reaction evaluations that are composed almost entirely (or only) 
of open-ended questions, 

3. Using unbalanced questions, which predominantly ask for positive or 
negative comments on the program, 

4. Failing to maximize this opportunity to collect useful information, and 

5. Allowing insufficient time for participants to complete reaction 
evaluations. 

Developing and Using Reaction Evaluation Effectively 

Research on reaction evaluation identifies a set of common guidelines for 
developing and using reaction evaluation effectively. The purposes of reaction evaluation 
must be determined (Robinson & Robinson, 1989; Zemke, 1996; Kirkpatrick, 1998). 
Purposes represent what the evaluator wants to know and the decisions one wants to 
make as a result of the information collected. Reaction evaluation includes the 
commonly asked questions about the program itself such as the facilities, schedule, 
handouts, and so forth. It is also important to determine how motivated the participants 
are to apply what they have learned, if anything will make it difficult for the participants 
to use their new skills on their job, what aspects of the program were most helpful, what 
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aspects of the program were least helpful, and if the participants value what they have 
learned. Questions developed for the reaction level evaluation must measure the course, 
content, 

instructor, and relevancy to the job. These are four areas considered essential to 
successful training programs. 

A form that will quantify reactions but also encourage written comments and 
suggestions should be designed (Robinson & Robinson, 1989; Zemke, 1996; Kirkpatrick, 

1 998). The ideal form provides the maximum amount of information and requires the 
minimum amount of time to complete. Reaction forms should permit both essay and 
scaled responses. Some part or section of the evaluation should focus directly on the 
learning and outcomes desired from the specific program. This specific focus prevents 
the “one size fits all” evaluation mentality. The form and delivery of the reaction 
evaluation must communicate a link between quality, process improvement, and action. 
Participants must be made to feel as though their individual response is a factor in the 
continuous improvement process. 

The evaluation process should ensure complete and immediate response 
(Robinson & Robinson, 1989; Parry, 1997 Kirkpatrick, 1998). To ensure complete 
response, reaction sheets should be completed during the training session. It is important 
to allow sufficient time at the end of the program so that participants are completing the 
evaluation on company time, not their own. The targeted time for completion of reaction 
sheets is 1 0 to 15 minutes. 

The evaluation process should ensure honest responses (Kirkpatrick, 1977; 
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Robinson & Robinson, 1989; Kirkpatrick, 1998). Much debate is held on the issue of 
requiring participants to sign their evaluations, commonly referred to as the “evidence vs. 
proof’ debate. If participants must sign their reaction sheets and they have a fear of being 
critical, then the reactions are only evidence of the feelings of the participants. If 
participants are completely candid and honest, then the reaction sheets are proof of the 
feelings and satisfaction of the participants. Kirkpatrick (1977) suggests that reaction 
sheets should not be signed and should be collected in such a way that there is no way to 
identify the person who completed them. The factor of honesty is one that is readily 
controllable and therefore proof of reaction is relatively easy to get. Another suggestion 
would be to make the signature optional and leave that decision to the participant. 

Acceptable standards of performance should be developed (Zemke, 1996; 
Kirkpatrick, 1998). Interpretation of the data generated by the reaction sheet is often 
complicated by the lack of performance standards against which to compare the results of 
specific programs. Standards of acceptable performance can be based on a realistic 
analysis of what can be expected considering such conditions as budgets, facilities 
available, skilled instructors available, and so on. 

Reliability of results can be increased by asking the participant’s supervisor to 
complete similar reaction sheets (Mikulecky & Lloyd, 1993; Zemke, 1996; Garavaglia, 
1998; Luketich, 1998). An abbreviated or slightly modified version of the reaction sheet 
can be used to measure the supervisor’s impressions and observations of the extent to 
which participants from their unit have changed after the program. This version allows 
another perspective on the information gathered directly from the participants without 
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costing a great deal in time and money. Supervisors are in excellent positions to provide 
data about participants’ strengths and weaknesses. Supervisors can also report changes in 
the duties and tasks participants perform on the job. Involving supervisors in the reaction 
evaluation process can also indicate a transfer of training (Level III). 

Duke Energy Corporation 

Duke Energy Corporation, an energy services company based in Charlotte, North 
Carolina, developed a Level I evaluation process which has become recognized as a 
model for the evaluation of workforce training and education (Allman, 1998; Kirkpatrick, 
1998). In 1994, Duke Energy sought to quantify the value of learning in the organization. 
Corporate training and education departments were asked to analyze and develop trends 
of the experiences in order to monitor continuous improvement of programs. 

Using Kirkpatrick’s model for evaluating training and education, an evaluation 
team began to work on developing a Level I evaluation process. Development and testing 
of the Level I instrument for evaluating participant reaction was conducted from 1 995 
until 1997. By the end of 1997, the number using the Level I reaction sheet grew to over 
25,000 participants. Analysis of the data began to reveal some very interesting trends 
(Allman, 1998). Allman states that training management came to the realization that, 

“the reaction to training and education is directly linked to the operation and business 
management aspects of the training unit” (p.136). Allman further comments that as the 
reactions of participants are continuously analyzed the trends suggest a direct correlation 
between reaction (Level I) and transfer to the job (Level III). 
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The case study of the Duke Energy Corporation’s training evaluation process 
suggests that the use of sophisticated techniques for analyzing participant reaction is 
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warranted (Allman 1998; Kirkpatrick, 1998). When used effectively, analysis of Level I 
evaluation data can help in the early detection of areas that need improvement or support 
the conclusion that a good result was achieved. 



Surveys have become a widely used and acknowledged research tool in most of 
the developed countries of the world (Rea & Parker, 1997). At the heart of survey 
research is the questionnaire development process. According to Rea and Parker, one of 
the preludes to the development of survey questions is the assembly of teams of content 
experts. These individuals are deemed to have knowledge of or interest in the issues 
associated with the research study. These teams, commonly referred to as focus groups, 
contribute significantly to an understanding of the key substantive issues necessary for the 
development of the questionnaire. Once these key issues have been outlined and 
specified, the researcher can prepare a draft questionnaire or survey instrument. 

Researchers should conduct a pretest of the questionnaire before using it in a 
study (Robinson & Robinson, 1989; Ary, Jacobs, & Razavieh, 1996; Borg, Gall, & Gall, 
1996; McMillan & Schumacher, 1997; Rea & Parker, 1997). Rea and Parker define a 
pretest as a small-scale implementation of the draft questionnaire that assesses such 
critical factors as the following: 

1 . Questionnaire clarity: Are the questions understood by the respondents? 



Developing and Administering a Questionnaire 
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2. Questionnaire comprehensiveness: Are the questions and response choices 
sufficiently comprehensive to cover a reasonably complete range of alternatives? 

3. Questionnaire acceptability: Are there any potential problems which must 
be identified and addressed by the researcher such as excessive questionnaire length or 
questions that are perceived to invade the privacy of the respondents? 

The pretest should include a sample of individuals from the population from 
which the researcher plans to draw respondents. The pretest form of the questionnaire 
should provide ample space for respondents to make criticisms and to make 
recommendations for improving the questionnaire (Borg et al., 1996; McMillan & 
Schumacher, 1997). According to Ary et al. (1996), observations made of the 
respondents as they complete the questionnaire can also be enlightening. Spending an 
undue amount of time on an item or leaving an item blank and returning to it later can 
indicate that there is a problem with that item. 

Following the pretest, the researcher must revise the questions as needed. The 
results of the pretest can be used to clarify the items or to eliminate some. The researcher 
may choose to perform a further pretest if these revisions are extensive (Rea & Parker, 
1997). Otherwise, the final questionnaire can be drafted and prepared for implementation 
in the actual study. 

It is important to inform potential respondents about the purpose of the study in 
advance of the questionnaire itself. Rea and Parker (1997) suggest that five components 
be included in an introductory statement. The five components include: (a) the 
organization or agency conducting the study; (b) the objectives and goals of the study and 
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the significance of the results to the respondents; (c) the basis of sample selection; 

(d) the characteristics the respondent possesses that led to their inclusion in the sample; 
and (e) the assurance that the respondent’s participation is valued, that answers are 
neither correct nor incorrect, and that responses will be treated confidentially. 

The questionnaire should be as concise as possible while still covering the 
necessary range of subject matter required in the study. The purpose of being sensitive to 
questionnaire length is to make certain that the questionnaire is not so long and 
cumbersome to the respondent that is engenders reluctance to complete the survey 
instrument (Rea & Parker, 1997). According to Rea and Parker, in-person surveys should 
ideally take less than 30 minutes. 



Summary 

Chapter II has reviewed the literature of research in areas related to the present 
study. The review has been subdivided by topic. First to be considered was research 
focused on the need for workforce training, both in the South and in Mississippi. 
Secondly, literature describing training program evaluation was reported. Research 
dealing with developing and administering questionnaires was presented for final 
consideration. Chapter III will present the design of the study. Chapter IV will present 
the findings of the study and Chapter V will present a discussion of the findings and 




recommendations. 



CHAPTER III 



METHODOLOGY 

The present study was conducted in two phases. Phase I was the development of 
the instrument for recording the reactions of the participants and their supervisors toward 
the effectiveness of the completed workforce training program. Once the instrument was 
developed and its content validity had been established, it was piloted in one community 
college district. Necessary revisions were made as a result of the pilot study. Phase II 
consisted of field testing the instrument in the districts of Mississippi Delta Community 
College and Coahoma Community College. Phase II results were used to compare the 
responses of the participants and their supervisors. 

Phase I 

Developing the Initial Instrument 

Nancy D. Padak and Gary M. Padak’s (1991) three categories for determining 
program effectiveness were used in the development of the instrument. These categories 
included: (a) programmatic, consisting of program structure and content; (b) personal, 
consisting of academic achievement and quality of life; and (c) external, consisting of 
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context-based and financial. Phase I began with a solicitation of expert opinions on the 
areas to be specified under each of the three effectiveness categories. 

The panel of experts included the directors, workforce specialists, basic skills 
specialists, and industrial coordinators from the One-Stop Career Centers at Mississippi 
Delta Community College and Coahoma Community College. The panel also included 
previous participants of workforce training programs and supervisors. The researcher 
personally met with these individuals to gain their input as to the content of the 
instrument. Prospective alternative-response, or agree-disagree, type items were selected 
for the proposed instrument by the researcher based on the literature review (see 
Appendix B). McMillan and Schumacher (1997) indicate that the forced choice format 
may lead to some frustration by the respondent. However, this format has merit in 
instances in which the respondents may have a tendency to cluster evidences in the 
middle category. 

Borg and Gall (1 996) suggest that questionnaires should be kept as short as 
possible. This was considered during the development of the instrument. The panel of 
experts had the opportunity to add and/or to revise items, as well as to comment on the 
appropriateness of each item. Since some of the subjects may have lower level literacy 
skills, the proposed instrument was designed to be administered by the researcher reading 
the items to the participants or the participants reading the items silently. The Fry 
Readability Graph (Bums, Roe, & Stoodt, 1998) was used to determine the instructional 
reading level of the pilot test instrument. 
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Revisions, additions, and/or comments were summarized and incorporated into 
the preliminary instrument. The revised instrument was returned to the panel of experts 
for final revisions. Minor revisions were incorporated into the initial instrument to be 
used in the pilot test (see Appendix C). A single form of the instrument was used. 
Participants were instructed to indicate whether they were a workforce training program 
participant or a supervisor of a workforce training program participant. 

The instrument was designed to measure the reaction of both the participants and 
their supervisors toward the effectiveness of the workforce training program. Level I of 
Kirkpatrick’s model provided the framework for the instrument. Evaluation based on the 
other three levels of Kirkpatrick’s model was left for future research studies. 

Establishing the Validity of the Instrument 

Content validity is determined by content experts who define the content that the 
instrument is assumed to represent and determine how well the content area is sampled by 
the items (Borg & Gall, 1 996). The panel of experts used to develop the instrument for 
this study included practitioners of workforce training programs, previous workforce 
training participants, and their supervisors. Their input was of significant value as they 
are the ones who design, implement, participate in, and evaluate the workforce training 
programs the One-Stop Career Centers deliver (Borg & Gall, 1996). Content validity of 
the instrument was obtained through the use of this expert panel. 
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Sample Selection for Pilot Test 

Convenience sampling was used for the purposes of this study. That is, the 
researcher selected subjects based on expedience and accessibility (McMillan & 
Schumacher, 1997). With the assistance of the One-Stop Career Center director at 
Mississippi Delta Community College, a recently completed workforce training program 
was selected. 

The cooperation of plant management was obtained by the researcher and the 
Center director. The subjects were employees who were current or previous participants 
of the selected workforce training program. Also included in the sample was the 
immediate supervisor of each employee. 

Pilot Test 

Borg and Gall (1996) assert that questionnaires should be piloted before using the 
instrument in any research study. They also suggest that the pilot form of the instrument 
should provide space for subjects to make criticisms and recommendations for improving 
the items. According to Borg and Gall, another useful strategy is to allow the subjects to 
write in their own words what each item means. 

The pilot test for this instrument was conducted for the purpose of analyzing each 
item. Subjects were expected to critique each item and make recommendations for 
improvement. The pilot test occurred with two separate groups: (a) the employees and 
(b) the supervisors. With the assistance of the company’s Human Resources Director, 
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convenient times to administer the instrument were identified. Selected employees and 
their supervisors were notified by the company’s Human Resources Director of the date 
and time scheduled for their appointment. 

After a brief introduction, the researcher read aloud a consent form which required 
the subject’s understanding of the researcher’s purposes for conducting the pilot test (see 
Appendix D). The subjects were asked to respond by giving an item critique and making 
recommendations. The subjects were told to note any areas of confusion which may 
cause the item to be misinterpreted by future participants. 

Results from the pilot test were used by the researcher to make any necessary 
revisions to the items. Items were revised and retested until they were understood 
accurately by all or most of the members of the pilot test sample (Borg & Gall, 1996). 
These items were included on the Field Test Survey Questionnaire (see Appendix E). 

Once the instrument was finalized, Phase II of the study began. 

Phase II 

Field Test 

Field testing of the instrument occurred with a convenience sample of two 
recently completed workforce training programs, one from the district of Mississippi 
Delta Community College and one from the district of Coahoma Community College. 

The subjects were employees and current or previous participants of the selected 
workforce training programs. Also included in the sample were the immediate 
supervisors of the employees. 
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The field test occurred with four groups during conveniently scheduled times: (a) 
participants in the Mississippi Delta Community College workforce training program, 

(b) supervisors of the participants in that training program, (c) participants in the 
Coahoma Community College workforce training program, and (d) supervisors of the 
participants in that training program. 

Data Analysis 

Upon completion of the field test of the instrument, a comparison of the 
descriptive data was made. The following questions were considered during data 
analysis: 

1 . Was there a difference between the reactions of participants and the 
reactions of the participants’ supervisors to the programmatic effectiveness 
of the recently completed workforce training program? 

2. Was there a difference between the reactions of participants and the 
reactions of the participants’ supervisors to the personal effectiveness of 
the recently completed workforce training program? 

3. Was there a difference between the reactions of participants and the 
reactions of the participants’ supervisors to the external effectiveness of 
the recently completed workforce training program? 

4. What conclusions can be made as to the effectiveness of workforce 
training programs using this instrument? 

For the purposes of this study, results were presented as descriptive data for each 
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item, including frequency of response and percentage of response. The researcher 
performed a comparative analysis of the responses from the participants, the supervisors, 
and the participant-supervisor pairs. 



Summary 

This chapter has described Phases I and II of the study. Phases I and II included 
the development of the evaluation instrument, the establishment of the validity of the 
instrument, the selection of the sample, the pilot test of the instrument, the field test of the 
instrument, and the statistical procedures which were used to analyze the data. The 
results of the study will be presented in Chapter IV. Chapter V will present a discussion 
of the results and recommendations for future studies. 
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CHAPTER IV 



RESULTS 
Phase I 

The present developmental study was conducted in two phases. Phase I was the 
development of the instrument for recording the reactions of the participants and their 
supervisors toward the effectiveness of the completed workforce training program. 

Establishing Content Validity 

Two panels of experts from local community college One-Stop Career Centers 
were used to develop and establish the content validity of the proposed instrument. The 
first panel consisted of seven members from the staff of Coahoma Community College’s 
One-Stop Career Center, a workforce training program instructor, and a previous 
workforce training program participant. The staff of Coahoma Community College’s 
One-Stop Career Center had a combined total of more than 33 years of experience in 
workforce training. The second panel consisted of 10 members from the staff of 
Mississippi Delta Community College’s One-Stop Career Center. The staff of this Center 
had a combined total of more than 94 years of experience in workforce training. 
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The researcher personally met with each expert panel at their One-Stop Career 
Center. The researcher presented an overview of the present study. A description of 
Nancy D. Padak and Gary M. Padak’s (1991) program effectiveness categories was 
discussed as the basis for the development of the items to be included on the instrument. 
The members of the panels critiqued prospective questionnaire items which were initially 
selected by the researcher based on the literature review (see Appendix B). The panels 
also contributed additions, revisions, and comments pertaining to each prospective item. 

The panels of experts made the following recommendations for the proposed 
instrument: 

1 . Order the sections on the questionnaire in the following manner: 
programmatic factors first, personal factors second, and external factors third. This 
placed the items in the order of those factors which could be assessed or observed most 
immediately following the workforce training program. 

2. Replace the word “teacher” with the word “instructor” in all items. Place 
these items in a separate category under programmatic factors entitled “Instructor.” 

3. Add one item distinguishing whether the training was mandatory or 
voluntary for the participant. The experts cautioned that the results from the instrument 
may be significantly different based on the response to this item. 

4. Remove subject specific items, such as those pertaining to math skills and 
reading job information, from the questionnaire. Removal of these items would allow the 
questionnaire to be used in evaluating all types of workforce training programs. 
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Pilot Test Questionnaire 

Revisions, additions, and comments from the two expert panels were incorporated 
by the researcher into the preliminary instrument. The revised instrument was returned to 
the panels for final revisions (see Appendix C). The panels of experts made no further 
revisions. 

The Fry Readability Graph (Bums, Roe, & Stoodt, 1998) was used to measure the 
readability of the pilot test questionnaire. The researcher determined the approximate 
instructional reading level of the pilot test questionnaire to be ninth grade. A participant 
consent form, to be used for the pilot and field tests, was developed by the researcher and 
approved by Delta State University’s Academic Research Coordinator. 

The pilot test questionnaire was developed as one single form to be used by the 
workforce training program participants and their supervisors. Each item was written to 
be read by the researcher in two different ways: (a) one for the workforce training 
program participants and (b) one for their supervisors. The pilot test questionnaire 
provided space following each item for subjects to make criticisms and recommendations 
for improving the items. 

Pilot Test 

With the assistance of the staff from Mississippi Delta Community College’s 
One-Stop Career Center, a manufacturer of healthcare products was selected as the pilot 
test site. The company’s Human Resources Director assembled a group of previous 
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workforce training program participants and supervisors as the pilot test group. The pilot 
test group consisted of 18 subjects: (a) 1 1 workforce training program participants and 
(b) 7 supervisors. 

After a brief introduction, the researcher read aloud the participant consent form 
(see Appendix D). All subjects indicated their understanding of the researcher’s purposes 
for conducting the pilot test. All subjects agreed to voluntarily participate in the pilot test 
by signing and dating the participant consent form. 

The researcher read aloud the directions printed on the pilot test survey 
questionnaire. Each subject indicated on their questionnaire whether they were a 
workforce training program participant or a supervisor of a workforce training program 
participant. The researcher then read aloud each item in two ways: (a) one for the 
workforce training program participants and (b) one for their supervisors. Subjects gave 
item critiques and made recommendations. The researcher noted areas of confusion 
indicated by the subjects. 

There were very few recommendations made by the subjects of the pilot test. 

Most of those recommendations included word or phrase changes. A suggestion was 
made to include a section for comments at the end of the survey questionnaire. The 
suggested item revisions were made by the researcher. 

The supervisors in the pilot test group indicated that during the actual 
administration of the survey questionnaire some may have difficulty completing the items 
in the personal factors and programmatic factors sections. Since the supervisors are 
asked to complete the survey questionnaire on the basis of the effectiveness of the 
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training program for their employees, these items may be difficult for them to evaluate, 
especially if they did not participate in the workforce training program. 

Phase II 

Phase II consisted of field testing the instrument in the districts of Mississippi 
Delta Community College and Coahoma Community College. Phase II results were used 
to compare the responses of the workforce training program participants and their 
supervisors. 

Field Test Questionnaire 

Suggested item revisions from the pilot test were made by the researcher. The 
items were then written in alternative-response, or agree-disagree, format for the field test 
questionnaire (see Appendix E). A total of 27 items was included on the field test 
questionnaire. The expected time of completion of the questionnaire was no more than 
20 minutes. 

Field testing of the instrument occurred with a convenience sample of two 
recently completed workforce training programs, one from the district of Mississippi 
Delta Community College and one from the district of Coahoma Community College. 
Subjects were employees and participants of the selected workforce training programs. 
Also included in the sample were the immediate supervisors of the employees. 
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Field Test I 

The first field test took place at a manufacturing facility of healthcare products 
located in the district of Mississippi Delta Community College. The selected training 
program was a recently completed voluntary basic mathematics course. The field test 
sample population included 1 1 participants in the workforce training program and their 
supervisors. There was a total of six supervisors, with three of them supervising more 
than one participant. 

The survey questionnaire was administered by the researcher to the participants in 
the training program at the end of the final training session. The questionnaire was 
administered to the supervisors of the participants two days later. All subjects agreed to 
voluntarily participate in the field test by signing and dating the participant consent form 
(see Appendix F). Each employee was assigned a number and their supervisor was 
assigned the same number. These numbers were written at the top of the field test survey 
questionnaire. This system allowed the researcher to match the pairs of participants and 
supervisors for data analysis. 

The researcher read aloud the directions printed on the field test survey 
questionnaire. Each subject indicated on their questionnaire whether they were a 
workforce training program participant or a supervisor of a workforce training program 
participant. The researcher then read aloud each item. Subjects indicated whether they 
agreed with or disagreed with each item. 
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Field Test II 

The second field test took place at a hospital located in the district of Coahoma 
Community College. The selected training program was a recently completed mandatory 
customer service course. The field test sample population included 23 participants in the 
workforce training program and their supervisors. There was a total of five supervisors, 
with each of them supervising more than one participant. 

The survey questionnaire was administered by the researcher to the participants in 
the training program at the end of the final training session. The questionnaire was 
administered to the supervisors of the participants one week later. All subjects agreed to 
voluntarily participate in the field test by signing and dating the participant consent form. 
Each employee was assigned a number and their supervisor was assigned the same 
number. These numbers were written at the top of the field test survey questionnaire. 
This system allowed the researcher to match the pairs of participants and supervisors for 
data analysis. 

The researcher read aloud the directions printed on the field test survey 
questionnaire. Each subject indicated on their questionnaire whether they were a 
workforce training program participant or a supervisor of a workforce training program 
participant. The researcher then read aloud each item. Subjects indicated whether they 
agreed with or disagreed with each item. 
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Analysis of Data 

Data from the field tests are presented in the following three sections: (a) results 
from Field Test I, (b) results from Field Test II, and (c) combined results of responses of 
participant-supervisor pairs from both field tests. 

Results from Field Test I 

Table 1 presents item responses of the participants, their supervisors, and 
participant-supervisor pairs by frequency and percent for Field Test I. Those items which 
were not answered by 100 percent of the test group were noted. 

Items 1 through 1 3 on the field test survey questionnaire assessed the 
programmatic aspects of the training program. Programmatic aspects included program 
content, program structure, facilities, and the instructor. Item responses by frequency and 
percent indicated three results: (a) consensus of response among the responding 
participants, (b) consensus of that same response among the responding supervisors, and 
(c) consensus of that same response among responding participant-supervisor pairs. 

Items 14 through 21 on the field test survey questionnaire assessed personal 
factors associated with the training program. Personal factors included quality of life and 
academic achievement. Item responses for items 14, 16, and 17 indicated results similar 
to those for items 1 through 13. 

Item responses for items 15, 18, 19, and 20 indicated that a majority of the 
participants agreed with the statements but a majority of the responding supervisors 




59 



49 



Table 1 

Item Responses by Frequency and Percent for Field Test I 

(N=l 1) 

A=Agree; D=Disagree 



Item 



Participant Supervisor Part./Sup. Pairs 



1 . 


The course material was organized 


A 


11(100) 


A 


7*(64) 


A 


7(64) 




and easy to follow. 


D 


0 


D 


0 


D 


0 


2. 


The information taught in the class 


A 


10*(91) 


A 


7*(64) 


A 


7(64) 




helped me in my personal life. 


D 


0 


D 


0 


D 


0 


3. 


The information taught in the class 


A 


8(73) 


A 


6*(55) 


A 


4(36) 




helped me know more about my 
current job. 


D 


3(27) 


D 


2 (18) 


D 


0 


4. 


The information taught in the class 


A 


7(64) 


A 


8*(73) 


A 


4(36) 




relates to my current job 
responsibilities. 


D 


4(36) 


D 


1 (9) 


D 


0 


5. 


Getting information about the classes 


A 


10(91) 


A 


10*(91) 


A 


9(82) 




offered at work was easy. 


D 


1 (9) 


D 


0 


D 


0 


6. 


The amount of time scheduled for 


A 


9(82) 


A 


4*(36) 


A 


4(36) 




this class was exactly what was 
needed to meet my objectives. 


D 


2(18) 


D 


0 


D 


0 


7. 


The facilities for the class were 


A 


11(100) 


A 


8*(73) 


A 


8(73) 




comfortable. 


D 


0 


D 


0 


D 


0 


8. 


I would recommend this course to 


A 


11(100) 


A 


5*(45) 


A 


5(45) 




others. 


D 


0 


D 


0 


D 


0 


9. 


My attendance at this training 


A 


2(18) 


A 


0 


A 


0 




program was mandatory. 


D 


9(82) 


D 


11(100) 


D 


9(82) 



(Table continued on next page) 
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Table 1 continued 



Item 



Participant Supervisor Part./Sup. Pairs 



10. 


I felt comfortable asking the 


A 


10*(91) 


A 


6*(55) 


A 


6(55) 




instructor questions. 


D 


0 


D 


0 


D 


0 


11. 


The instructor encouraged my 


A 


11 (100) 


A 


6*(55) 


A 


6(55) 




participation in class. 


D 


0 


D 


0 


D 


0 


12. 


The instructor was available when 


A 


10(91) 


A 


6*(55) 


A 


6(55) 




I needed help. 


D 


1 (9) 


D 


0 


D 


0 


13. 


The instructor presented the 


A 


11(100) 


A 


6*(55) 


A 


6(55) 




information in a clear, 
understandable, and professional 
manner. 


D 


0 


D 


0 


D 


0 


14. 


I have more confidence in 


A 


10(91) 


A 


6*(55) 


A 


5(45) 




performing my job responsibilities. 


D 


1 (9) 


D 


2 (18) 


D 


0 


15. 


My attitude about work has 


A 


10(91) 


A 


3 *(27) 


A 


3(27) 




improved. 


D 


1 (9) 


D 


7 (64) 


D 


1(9) 


16. 


I am proud of my work and how it 


A 


10(91) 


A 


9*(82) 


A 


8(73) 




contributes to the success of the 
company. 


D 


1 (9) 


D 


1 (9) 


D 


0 


17. 


My desire to continue my education 


A 


10(91) 


A 


8*(73) 


A 


7(64) 




has increased. 


D 


1 (9) 


D 


2 (18) 


D 


0 


18. 


My ability to communicate with my 


A 


10(91) 


A 


4*(36) 


A 


3(27) 




co-workers has improved. 


D 


1 (9) 


D 


5 (45) 


D 


0 


19. 


My ability to communicate with my 


A 


8(73) 


A 


5 (45) 


A 


4(36) 




supervisor has improved. 


D 


3(27) 


D 


6 (55) 


D 


2(18) 


20. 


My ability to solve problems and 


A 


9(82) 


A 


4*(36) 


A 


4(36) 




make decisions, large or small, on 
my job has increased. 


D 


2(18) 


D 


5 (45) 


D 


2(18) 




(Table continued on next page) 
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Table 1 continued 



Item 



Participant Supervisor Part./Sup. Pairs 



21. 


I am now able to perform my 


A 


10(91) 


A 


6(55) 


A 


5(45) 




job better. 


D 


1 (9) 


D 


5(45) 


D 


0 


22. 


My chances of getting a job, 


A 


10(91) 


A 


6(55) 


A 


5(45) 




keeping my job, or advancing in 
my job have increased. 


D 


1 (9) 


D 


5(45) 


D 


0 


23. 


My chances of making more money 


A 


8(73) 


A 


1*(9) 


A 


1(9) 




have increased. 


D 


3(27) 


D 


6(55) 


D 


3(27) 


24. 


My ability to contribute to the 


A 


9(82) 


A 


4*(36) 


A 


3(27) 




well-being of my family has 
increased. 


D 


2(18) 


D 


1 (9) 


D 


1(9) 


25. 


I have been given or offered more 


A 


5(45) 


A 


1* (9) 


A 


0 




responsibility in my job. 


D 


6(55) 


D 


6(55) 


D 


4(36) 


26. 


I get along better with my 


A 


8(73) 


A 


2*(18) 


A 


1(9) 




co-workers. 


D 


3(27) 


D 


5 (45) 


D 


1(9) 


27. 


I get along better with my 


A 


8(73) 


A 


3*(27) 


A 


2(18) 




supervisor. 


D 


3(27) 


D 


5 (45) 


D 


1(9) 



Note, indicates items not answered by 100% of test group. 
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disagreed with the statements. Item 15, “my attitude about work has improved” was 
contained in the quality of life item category. A total of 10 (91%) of the participants 
agreed with the statement indicating an improvement in their attitude about work as a 
result of the training program. In contrast, only three (27%) of the supervisors agreed 
with the statement and seven (64%) of the supervisors disagreed with the statement. A 
total of four (36%) participant-supervisor pairs were in agreement, which was the 
maximum number possible based on the responses of each group. 

Items 18, 19, and 20 were contained in the category of academic achievement. Item 
responses for those items showed disagreement in responses among a majority of the 
participants and a majority of the responding supervisors. Items 1 8 and 1 9 referred to an 
improvement in communication with co-workers and supervisors, respectively. A 
majority of the participants, 10 (91%) and 8 (73%), agreed with both statements, 
indicating an improvement in their communication with co-workers and supervisors as a 
result of the training program. In contrast, a majority of the responding supervisors 
disagreed with those statements, five (45%) and six (55%), respectively. A total of three 
(27%) participant-supervisor pairs were in agreement on item 1 8, and six (54%) pairs 
were in agreement on item 1 9. 

Item 20 referred to an improvement in the ability to solve problems and make 
decisions. A total of nine (82%) of the participants agreed with the statement, indicating 
an improvement in their ability to solve problems and make decisions as a result of the 
training program. A total of four (36%) of the responding supervisors agreed with the 
statement, whereas five (45%) of the supervisors disagreed with the statement. A total of 
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six (54%) participant-supervisor pairs were in agreement with the item, a majority of the 
respondent pairs and the maximum number possible based on the responses of each 
group. 

Items 22 through 27 on the field test survey questionnaire assessed external factors 
associated with the training program. External factors included financial and context- 
based categories of items. Item responses for items 22, 24, and 25 indicated consensus of 
the same response among the responding participants, supervisors, and participant- 
supervisor pairs. 

Item responses for items 23, 26, and 27, which assessed external factors, indicated 
that a majority of the participants agreed with the statements but a majority of the 
responding supervisors disagreed with the statements. Item 23, “my chances of making 
more money have increased” was contained in the financial category of items. A total of 
eight (73%) of the participants agreed with the statement, indicating an increase in their 
chances of making more money as a result of the training program. Only one (9%) of the 
supervisors agreed with the statement, whereas a total of six (55%) of the supervisors 
disagreed with the statement. A total of four (36%) participant-supervisor pairs were in 
agreement, which was the maximum number possible based on the responses of each 
group. 

Items 26 and 27, were contained in the context-based category of items. Context- 
based items referred to an improvement in the ability to get along with co-workers and 
supervisors, respectively. For both items, a majority of the participants agreed with the 
statement and a majority of the responding supervisors disagreed with the statement. A 
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total of eight (73%) of the participants agreed with both statements, indicating an 
improvement in their ability to get along with co-workers and supervisors as a result of 
the training program. In contrast, a total of five (45%) of the supervisors disagreed with 
both statements. Only two (1 8%) of the supervisors agreed with item 26 and only three 
(27%) of the supervisors agreed with item 27. Two (18%) participant-supervisor pairs 
were in agreement on item 26, and three (27%) pairs were in agreement on item 27. 

Results from Field Test II 

Table 2 presents item responses of the participants, their supervisors, and 
participant-supervisor pairs by frequency and percent for Field Test II. Those items 
which were not answered by 100 percent of the test group were noted. 

Items 1 through 1 3 on the field test survey questionnaire assessed the programmatic 
aspects of the training program. Programmatic aspects included program content, 
program structure, facilities, and the instructor. Item responses by frequency and percent 
indicated three results: (a) consensus of response among the responding participants, (b) 
consensus of that same response among the responding supervisors, and (c) consensus of 
that same response among responding participant-supervisor pairs. 

Items 14 through 21 on the field test survey questionnaire assessed personal factors 
associated with the training program. Personal factors included quality of life and 
academic achievement. Item responses for items 14, 15, 16 and 21 indicated results 
similar to those for items 1 through 13. 




65 



55 



Table 2 

Item Responses by Frequency and Percent for Field Test II 

(N=23) 

A=Agree; D=Disagree 



Item 



Participant Supervisor Part./Sup. Pairs 



1 . 


The course material was organized 


A 


23(100) 


A 


23(100) 


A 


23(100) 




and easy to follow. 


D 


0 


D 


0 


D 


0 


2. 


The information taught in the class 


A 


20(87) 


A 


8*(35) 


A 


7(30) 




helped me in my personal life. 


D 


3(13) 


D 


6 (26) 


D 


1(4) 


3. 


The information taught in the class 


A 


21(91) 


A 


23(100) 


A 


21(91) 




helped me know more about my 
current job. 


D 


2 (9) 


D 


0 


D 


0 


4. 


The information taught in the class 


A 


23(100) 


A 


23(100) 


A 


23(100) 




relates to my current job 
responsibilities. 


D 


0 


D 


0 


D 


0 


5. 


Getting information about the classes 


A 


18(78) 


A 


23(100) 


A 


23(100) 




offered at work was easy. 


D 


5(22) 


D 


0 


D 


0 


6. 


The amount of time scheduled for 


A 


20(87) 


A 


23(100) 


A 


20(87) 




this class was exactly what was 
needed to meet my objectives. 


D 


3(13) 


D 


0 


D 


0 


7. 


The facilities for the class were 


A 


22(96) 


A 


23(100) 


A 


22(96) 




comfortable. 


D 


1 (4) 


D 


0 


D 


0 


8. 


I would recommend this course 


A 


22(96) 


A 


23(100) 


A 


22(96) 




to others. 


D 


1 (4) 


D 


0 


D 


0 


9. 


My attendance at this training 


A 


22(96) 


A 


23(100) 


A 


22(96) 




program was mandatory. 


D 


1 (4) 


D 


0 


D 


0 




(Table continued on next page) 



56 



Table 2 continued 



Item 



Participant Supervisor Part./Sup. Pairs 



10. 


I felt comfortable asking the 


A 


23(100) 


A 


9*(39) 


A 


9(39) 




instructor questions. 


D 


0 


D 


0 


D 


0 


11. 


The instructor encouraged my 


A 


22(96) 


A 


23(100) 


A 


22(96) 




participation in class. 


D 


1(4) 


D 


0 


D 


0 


12. 


The instructor was available when 


A 


22 *(96) 


A 


23(100) 


A 


22(96) 




I needed help. 


D 


0 


D 


0 


D 


0 


13. 


The instructor presented the 


A 


23(100) 


A 


23(100) 


A 


23(100) 




information in a clear, 
understandable, and professional 
manner. 


D 


0 


D 


0 


D 


0 


14. 


I have more confidence in 


A 


21(91) 


A 


10*(43) 


A 


8(35) 




performing my job responsibilities. 


D 


2 (9) 


D 


2 (9) 


D 


0 


15. 


My attitude about work has 


A 


21(91) 


A 


10* (43) 


A 


9(39) 




improved. 


D 


2 (9) 


D 


2 (9) 


D 


0 


16. 


I am proud of my work and how it 


A 


23(100) 


A 


17*(74) 


A 


17(74) 




contributes to the success of the 
company. 


D 


0 


D 


1 (4) 


D 


0 


17. 


My desire to continue my education 


A 


22*(96) 


A 


1*(4) 


A 


1(4) 




has increased. 


D 


0 


D 


6(26) 


D 


0 


18. 


My ability to communicate with my 


A 


23(100) 


A 


9(39) 


A 


9(39) 




co-workers has improved. 


D 


0 


D 


14(61) 


D 


0 


19. 


My ability to communicate with my 


A 


20(87) 


A 


3(13) 


A 


3(13) 




supervisor has improved. 


D 


3(13) 


D 


20(87) 


D 


3(13) 


20. 


My ability to solve problems and 


A 


21(91) 


A 


3*(13) 


A 


3(13) 




make decisions, large or small, on 
my job has increased. 


D 


2 (9) 


D 


18 (78) 


D 


2 (9) 




(Table continued on next page) 



Table 2 continued 





Item 


Participant 


Supervisor 


Part./Sup. Pairs 


21. 


I am now able to perform my 


A 


22(96) 


A 


23(100) 


A 


23(100) 




job better. 


D 


1 (4) 


D 


0 


D 


0 


22. 


My chances of getting a job, 


A 


19(83) 


A 


5(22) 


A 


5(22) 




keeping my job, or advancing in 
my job have increased. 


D 


4(17) 


D 


18(78) 


D 


4(17) 


23. 


My chances of making more money 


A 


13(57) 


A 


0 


A 


0 




have increased. 




D 


10(43) 


D 


23(100) 
















D10(43) 


24. 


My ability to contribute to the 


A 


20(87) 


A 


5*(22) 


A 


5(22) 




well-being of my family has 
increased. 


D 


3(13) 


D 


17(74) 


D 


3(13) 


25. 


I have been given or offered more 


A 


10*(43) 


A 


7(30) 


A 


3(13) 




responsibility in my job. 


D 


12 (52) 


D 


16(70) 


D 


7(30) 


26. 


I get along better with my 


A 


22(96) 


A 


2 (9) 


A 


2 (9) 




co-workers. 


D 


1 (4) 


D 


21(91) 


D 


1 (4) 


27. 


I get along better with my 


A 


21(91) 


A 


3(13) 


A 


3(13) 




supervisor. 


D 


2 (9) 


D 


20(87) 


D 


2 (9) 



Note. *indi cates items not answered by 100% of test group. 



58 



Item responses for items 17 through 20 indicated that a majority of the responding 
participants agreed with the statements but a majority of the responding supervisors 
disagreed with the statements. Item 17, “my desire to continue my education has 
increased” was contained in the quality of life item category. A total of 22 (96%) of the 
participants agreed with the statement. In contrast, six (26%) of the supervisors disagreed 
with the statement and only one (4%) of the supervisors agreed with the statement. One 
(4%) participant-supervisor pair was in agreement, which was the maximum number 
possible based on the responses of each group. 

Items 18, 19, and 20 were contained in the category of academic achievement. Item 
responses for those items showed disagreement in responses among a majority of the 
participants and a majority of the responding supervisors. Items 1 8 and 19 referred to an 
improvement in communication with co-workers and supervisors, respectively. A 
majority of the participants, 23 (100%) and 20 (87%), agreed with both statements, 
indicating an improvement in their communication with co-workers and supervisors as a 
result of the training program. In contrast, a majority of the supervisors, 14 (61%) and 20 
(87%), disagreed with those statements. A total of nine (39%) participant-supervisor 
pairs were in agreement on item 1 8, and six (26%) pairs were in agreement on item 19. 

Item 20 referred to an improvement in the ability to solve problems and make 
decisions. A total of 2 1 (91 %) of the participants agreed with the statement, indicating an 
improvement in their ability to solve problems and make decisions as a result of the 
training program. A total of three (1 3%) of the responding supervisors agreed with the 
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statement, whereas 18 (78%) of the supervisors disagreed with the statement. A total of 
five (22%) participant-supervisor pairs were in agreement with the item, the maximum 
number possible based on the responses of each group. 

Items 22 through 27 on the field test survey questionnaire assessed external factors 
associated with the training program. External factors included financial and context- 
based item categories. Item responses for item 25 indicated consensus of the same 
response among the responding participants, supervisors, and participant-supervisor pairs. 
Item responses for items 22, 23, 24, 26, and 27, which assessed external factors, 
indicated that a majority of the responding participants agreed with the statements but a 
majority of the responding supervisors disagreed with the statements. Item 22, 23, and 24 
were contained in the financial category of items. Item 22 referred to an increase in the 
participant’s ability to get a job, keep a job, or advance in a job. Item 23 referred to an 
increase in the participant’s chances of making more money. Item 24 referred to an 
increase in the ability to contribute to the well-being of one’s family. 

A majority of the participants agreed with all three items in the financial category: 

19 (83%), 13 (57%), and 20 (87%), respectively. In contrast, a majority of the responding 
supervisors disagreed with all three statements: 18 (78%), 23 (100%), and 17 (74%), 
respectively. A total of nine (39%) participant-supervisor pairs were in agreement with 
item 22. A total of 10 (43%) participant-supervisor pairs were in agreement with item 23. 
A total of eight (35%) participant-supervisor pairs were in agreement with item 24. All 
three totals indicated the maximum number possible based on the responses of each 




group. 
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Items 26 and 27, were contained in the context-based category of items. Context- 
based items referred to an improvement in the ability to get along with co-workers and 
supervisors, respectively. For both items, a majority of the participants agreed with the 
statement and a majority of the supervisors disagreed with the statement. 

A total of 22 (96%) of the participants agreed with item 26, indicating an 
improvement in their ability to get along with co-workers as a result of the training 
program. In contrast, two (9%) of the supervisors agreed with the statement and 21 
(91%) disagreed with the statement. A total of 21 (91%) of the participants agreed with 
item 27, indicating an improvement in their ability to get along with their co-workers and 
supervisors as a result of the training program. In contrast, three (13%) of the supervisors 
agreed with the statement and 20 (87%) of the supervisors disagreed with both 
statements. Three (13%) participant-supervisor pairs were in agreement with item 26, 
and five (22%) pairs were in agreement with item 27. Both totals indicated the maximum 
number possible based on the responses of each group. 

Combined Results of Responses of Participant-Supervisor Pairs from Field Tests I and II 

Data from the combined item responses of participant-supervisor pairs for items 
evaluating programmatic factors was considered as it related to Research Question 1 : “Is 
there a difference between the reactions of participants and the reactions of the 
participants’ supervisors to the programmatic effectiveness of the recently completed 
workforce training program?” 
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Table 3 presents the combined item responses of participant-supervisor pairs by 
frequency and percent for programmatic factors related to the workforce training 
program. Programmatic factors included items related to program content, program 
structure, facilities, and the instructor. 

Items 1 through 4 assessed the effectiveness of the training program with respect to 
the program content. A majority of the participant-supervisor pairs were in agreement of 
response for items 1,3, and 4. The respective percentages of agreement for items 1,3, 
and 4 were 88%, 73%, and 79%. Item 1 had no disagreement of response among 
participant-supervisor pairs. The percentages of disagreement of response among 
participant-supervisor pairs for items 3 and 4 were very small. A total of six (18%) 
participant-supervisor pairs had a disagreement of response for item 3. A total of five 
(15%) participant-supervisor pairs had a disagreement of response for item 4. There were 
some supervisors that did not respond to items 1,3, and 4. The percentages of no 
complete response for these items were 12%, 9%, and 6%, respectively. 

The percentage of agreement of response among participant-supervisor pairs for 
item 2, “the information taught in the class helped me in my personal life,” was 44%. A 
total of 1 3 (38%) pairs did not respond completely to item 2. A total of six (18%) 
participant-supervisor pairs had a disagreement of response for item 2. 

Items 5 through 9 contained information about the program structure and facilities. 
A majority of the participant-supervisor pairs reached a consensus of response for items 5 
through 9. Items 5 through 9 had 32 (94%), 24 (71%), 30 (88%), 27 (79%), and 3 1 
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Table 3 

Combined Item Responses of Participant-Supervisor Pairs 
by Frequency and Percent for Programmatic Factors 

(N=34) 

A= Agreement of response; D=Disagreement of response; N=No complete response 



Items 



Part/Sup. Pairs 



Program Content 



1 . 


The course material was organized and 


A 


30(88) 




easy to follow. 


D 


0 






N 


4(12) 


2. 


The information taught in the class helped 


A 


15(44) 




me in my personal life. 


D 


6(18) 






N 


13(38) 


3. 


The information taught in the class helped 




A25(73) 




me know more about my current job. 


D 


6(18) 






N 


3(9) 


4. 


The information taught in the class relates to 


A 


27(79) 




my current job responsibilities. 


D 


5(15) 






N 


2(6) 


Program Structure and Facilities 






5. 


Getting information about the classes offered at 


A 


32(94) 




work was easy. 


D 


1(3) 






N 


1(3) 


6. 


The amount of time scheduled for this class was 


A 


24(71) 




exactly what was needed to meet my objectives. 


D 


3(9) 






N 


7(20) 



(Table continued on next page) 
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Table 3 continued 





Items 


Part./Sup. Pairs 


7. 


The facilities for the class were comfortable. 


A 


30(88) 






D 


1(3) 






N 


3(9) 


8. 


I would recommend this course to others. 


A 


27(79) 






D 


1(3) 






N 


6(18) 


9. 


My attendance at this training program was 


A 


31(91) 




mandatory. 


D 


3(9) 






N 


0 


Instructor 






10. 


I felt comfortable asking the instructor questions. 


A 


15(44) 






D 


0 






N 


19(56) 


11. 


The instructor encouraged my participation in class. 


A 


28(82) 






D 


1(3) 






N 


5(15) 


12. 


The instructor was available when I needed help. 


A 


28(82) 






D 


0 






N 


6(18) 


13. 


The instructor presented the information in a 


A 


29(85) 




clear, understandable, and professional manner. 


D 


0 






N 


5(15) 
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(91%) of its respondent pairs in agreement. The percentages of disagreement among 
participant-supervisor pairs for items 5 through 9 were very small. Items 5, 7, and 8 had 
only one (3%) participant-supervisor pair with a disagreement of response. Items 6 and 9 
had three (9%) participant-supervisor pairs with a disagreement of response. Some 
supervisors did not respond to items 5 through 8. The percentages of no complete 
response for items 5 through 8 were 3%, 20%, 9%, and 18%. All participant-supervisor 
pairs responded completely to item 9. 

Items 10 through 13 assessed the effectiveness of the training program with respect 
to the instructor. A majority of the participant-supervisor pairs reached a consensus of 
response for items 1 1 through 13. A total of 28 (82%) pairs reached a consensus of 
response for items 1 1 and 12. A total of 29 (85%) pairs reached a consensus of response 
for item 13. Item 1 1, “the instructor encouraged my participation in class,” was the only 
item which had a disagreement of response among participant-supervisor pairs. One 
(3%) pair had a disagreement of response for item 11. Items 1 1 through 13 had a low 
percentage of no response among participant-supervisor pairs. A total of five (15%) 
participant-supervisor pairs did not respond completely to items 1 1 and 13. Six (1 8%) 
participant-supervisor pairs did not respond completely to item 12. 

A total of 15 (44%) pairs reached a consensus of response for item 10. A total of 19 
(56%) pairs did not respond completely to item 10. There was no disagreement of 
response among participant-supervisor pairs for item 10. 

Based on these data, Research Question 1 must be answered negatively. There was 
no difference between the reactions of participants and the reactions of the participants’ 
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supervisors to the programmatic effectiveness of the recently completed workforce 
training program. Only one of the 1 3 items evaluating programmatic effectiveness, item 
1 0, did not have a majority of respondent pairs with a consensus of response. The 
remaining twelve items had a majority of consensus of response among the participant- 
supervisor pairs. Therefore, there was no difference between the reactions of participants 
and the reactions of the participants’ supervisors to the programmatic effectiveness of the 
workforce training program. 

Data from the combined item responses of participant-supervisor pairs to items 
evaluating personal factors was considered as it related to Research Question 2: “Is there 
a difference between the reactions of participants and the reactions of the participants’ 
supervisors to the personal effectiveness of the recently completed workforce training 
program?” 

Table 4 presents the combined item responses of participant-supervisor pairs by 
frequency and percent for personal factors related to the workforce training program. 
Personal factors included items related to the participant’s quality of life and academic 
achievement. 

Items 14 through 17 assessed the effectiveness of the workforce training program 
with respect to the participant’s quality of life. Analysis of responses from participant- 
supervisor pairs for those items showed item 16 as the only item with a majority of the 
participant-supervisor pairs in agreement of response. Item 16, “I am proud of my work 
and how it contributes to the success of the company” had 25 (73%) of the participant- 
supervisor pairs with an agreement of response. A total of three (9%) of the 
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Table 4 

Combined Item Responses of Participant-Supervisor Pairs 
by Frequency and Percent for Personal Factors 

(N=34) 

A=Agreement of response; D=Disagreement of response; N=No complete response 



Items 


Part./Sup. Pairs 


Quality of Life 


14. I have more confidence in performing my 


A 


13(38) 


job responsibilities. 


D 


7(21) 




N 


14(41) 


15. My attitude about work has improved. 


A 


13(38) 




D 


9(27) 




N 


12(35) 


16. Iam proud of my work and how it contributes 


A 


25(73) 


to the success of the company. 


D 


3(9) 




N 


6(18) 


1 7. My desire to continue my education has increased. 


A 


8(24) 




D 


0 




N 


26(76) 


Academic Achievement 


1 8. My ability to communicate with my co-workers 


A 


12(35) 


has improved. 


D 


20(59) 




N 


2(6) 


1 9. My ability to communicate with my supervisor 


A 


12(35) 


has improved. 


D 


22(65) 




N 


0 




(Table continued on next page) 
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Table 4 continued 



Items 



Part./Sup. Pairs 



20. My ability to solve problems and make decisions, 


A 


11(32) 


large or small, on my job has increased. 


D 


19(56) 




N 


4(12) 


21 . I am now able to perform my job better. 


A 


28(82) 




D 


6(18) 




N 


0 
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participant-supervisor pairs were in disagreement of response to this item. Six (18%) of 
the participant-supervisor pairs did not respond completely to item 16. Analysis of 
responses to item 1 5, “my attitude about work has improved,” showed approximately the 
same percentage of agreement of response and no complete response among participant- 
supervisor pairs, 38% and 35% respectively. 

Item responses for items 14 and 17 showed a high percentage of no complete 
response among participant-supervisor pairs. A total of 14 (41 %) of the participant- 
supervisor pairs did not respond completely to item 14, “I have more confidence in 
performing my job responsibilities.” A total of 26 (76%) of the participant-supervisors 
pairs did not respond completely to item 17, “my desire to continue my education has 
increased.” 

The percentage of consensus of response among participant-supervisor pairs for 
items 14 and 15 was 38 percent. The percentage of consensus of response among 
participant-supervisor pairs for item 1 7 was 24 percent. The percentages of disagreement 
of response among participant-supervisor pairs for items 14, 15, and 1 7 were very low. 
Item 14 had a total of seven (21%) participant- supervisor pairs with a disagreement of 
response. A total of nine (27%) of the participant-supervisor pairs were in disagreement 
of response to item 1 5. There were no participant-supervisor pairs in disagreement of 
response to item 17. 

Items 18 through 21 assessed the effectiveness of the workforce training program 
with respect to the academic achievement of the participant. Item responses from 
participant-supervisor pairs for items 1 8 through 20 showed a majority of disagreement of 
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response. Item 1 8, “my ability to communicate with my co-workers has improved,” 
had 20 (59%) pairs with a disagreement of response. Item 19, “my ability to 
communicate with my supervisor has improved,” had 22 (65%) pairs with a disagreement 
of response. Item 20, “my ability to solve problems and make decisions, large or small, 
on my job has increased,” had 19 (56%) pairs with a disagreement of response. A total of 
12 (35%) participant-supervisor pairs were in agreement of response to items 18 and 19. 

A total of 1 1 (32%) participant-supervisor pairs were in agreement of response to item 20. 
The percentages of no complete response to items 1 8 through 20 were very low. A total 
of two (6%) participant-supervisor pairs did not respond completely to item 18. All 
participant-supervisor pairs responded completely to item 19. A total of 4 (12%) 
participant-supervisor did not respond completely to item 20. 

Item 21, “I am now able to perform my job better,” was the only item in the 
category of academic achievement that had a majority of participant-supervisor pairs with 
an agreement of response. A total of 28 (82%) pairs had an agreement of response to 
item 21 . A total of six (18%) participant-supervisor pairs were in disagreement of 
response. All participant- supervisor pairs responded completely to item 21. 

Based on these data, Research Question 2 must be answered affirmatively. There 
was a difference between the reactions of participants and the reactions of the 
participants’ supervisors to the personal effectiveness of the recently completed 
workforce training program. Only two of the eight items evaluating personal 
effectiveness, item 16 and item 21, had a majority of respondent pairs with a consensus of 
response. Item responses for the remaining six items showed either a majority of 
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disagreement of response or a majority of no complete response. The item responses 
indicated a difference between the reactions of participants and the reactions of the 
participants’ supervisors to the personal effectiveness of the workforce training program. 

Data from the combined item responses of participant-supervisor pairs to items 
evaluating external factors was considered as it related to Research Question 3: “Is there 
a difference between the reactions of participants and the reactions of the participants’ 
supervisors to the external effectiveness of the recently completed workforce training 
program?” 

Table 5 presents the combined item responses of participant-supervisor pairs by 
frequency and percent for external factors related to the workforce training program; 
External factors included financial and context-based categories of items. 

Items 22 through 24 were contained in the financial category of items. Analysis of 
responses from participant-supervisor pairs to items 22 through 24 showed a majority of 
disagreement in response among the pairs. A total of 20 (59%) participant-supervisor 
pairs were in disagreement of response to item 22, “my chances of getting a job, keeping 
my job, or advancing in my job have increased.” A total of 16 (47%) participant- 
supervisor pairs were in disagreement of response to item 23, “my chances of making 
more money have increased.” A total of 1 5 (44%) participant-supervisor pairs were in 
disagreement of response to item 24, “my ability to contribute to the well-being of my 
family has increased.” A total of 14 (41%) of the participant-supervisor pairs reached a 
consensus of response to items 22 and 23. A total of 12 (35%) of the participant- 
supervisor pairs reached a consensus of response to item 24. The percentages of no 
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Table 5 

Combined Item Responses of Participant-Supervisor Pairs 
by Frequency and Percent for External Factors 

(N=34) 





Items 


Part./Sup. Pairs 


Financial 






22. 


My chances of getting a job, keeping my job, 


A 


14(41) 




or advancing in my job have increased. 


D 


20(59) 






N 


0 


23. 


My chances of making more money have increased. 


A 


14(41) 






D 


16(47) 






N 


4(12) 


24. 


My ability to contribute to the well-being of my 


A 


12(35) 




family has increased. 


D 


15(44) 






N 


7(21) 


Context-Based 






25. 


I have been given or offered more responsibility 


A 


14(41) 




in my job. 


D 


15(44) 






N 


5(15) 


26. 


I get along better with my co-workers. 


A 


5(15) 






D 


25(73) 






N 


4(12) 


27. 


I get along better with my supervisor. 


A 


8(23) 






D 


23(68) 






N 


3(9) 



Note. A= Agreement of response; D=Disagreement of response; N=No response. 
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complete response among the pairs to items 22 through 24 were veiy small. All 
participant-supervisor pairs responded completely to item 22. Four (12%) pairs did not 
respond completely to item 23 and seven (2 1 %) pairs did not respond completely to item 
24. 

Items 25 through 27 were contained in the context-based category of items. Item 
responses from participant-supervisor pairs to those items also showed a majority of 
disagreement in response among the pairs. A total of 1 5 (44%) participant-supervisor 
pairs were in disagreement of response to item 25, “I have been given or offered more 
responsibility in my job.” A total of 25 (73%) participant-supervisor pairs were in 
disagreement of response to item 26, “I get along better with my co-workers.” A total of 
23 (68%) participant-supervisor pairs were in disagreement of response to item 27, “I get 
along better with my supervisor.” A total of 14 (41%) pairs reached a consensus of 
response to item 25. Fewer participant-supervisor pairs reached a consensus of response 
to items 26 and 27, with 5 (15%) and 8 (23%) respectively. The percentages of no 
complete response among the pairs to items 25 through 27 were very small. Five (15%) 
of the participant-supervisor pairs did not respond completely to item 25. A total of four 
(12%) pairs did not respond completely to item 26. Three (9%) of the participant- 
supervisor pairs did not respond completely to item 27. 

Based on these data, Research Question 3 must be answered affirmatively. There 
was a difference between the reactions of participants and the reactions of the 
participants’ supervisors to the external effectiveness of the recently completed workforce 
training program. A majority of responding participant-supervisor pairs were in 




disagreement of response to all six items evaluating external factors. This indicated a 
difference between 

the reactions of participants and the reactions of the participants’ supervisors to the 
external effectiveness of the workforce training program. 

Summary 

Chapter IV has presented the results from the study. A discussion of the results, 
well as recommendations for future studies, will be found in Chapter V. 



CHAPTER V 



SUMMARY AND DISCUSSION 

The purpose of this study was to develop an evaluation instrument to assess the 
effectiveness of workforce training programs. The present developmental study was 
conducted in two phases. Phase I was the development of the instrument for recording 
the reactions of the participants and their supervisors toward the effectiveness of the 
completed workforce training program. Based on results reported in Chapter IV, the 
purpose of the study was achieved in Phase I. Phase II consisted of field testing the 
instrument. Phase II results were used to compare the responses of the workforce training 
program participants and the supervisors. Phase II also sought to determine whether there 
was agreement among participants and their supervisors toward the effectiveness of the 
workforce training program. 

Two panels of experts from local community college One-Stop Career Centers 
were used to develop and establish the content validity of the proposed instrument. 
Literature in the area of effective program evaluation consistently pointed to the 
importance of involving all stakeholders in the design and implementation of the 
evaluation (Tracey, 1968; Merwin, 1992; Mikulecky & Lloyd, 1993; Hilbert, Preskill, & 
Russ-Eft, 1997; Phillips, 1998; Boulmetis & Dutwin, 2000). The pilot test of the 
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instrument occurred at a manufacturing facility located in one of the local community 
college districts. The pilot test group consisted of 1 8 subjects: (a) 1 1 workforce training 
program participants and (b) 7 supervisors. 

Field testing of the instrument occurred at two separate locations, one in each 
community college district. A manufacturer of healthcare products was the site of the 
first field test. The training program was a voluntary basic mathematics course. The 
second field test took place at a regional hospital. The training program was a mandatory 
customer service course. The sample group for the field test consisted of 34 workforce 
training program participants and 1 1 supervisors. 

Data analysis procedures included an item analysis of responses of the 
participants, their supervisors, and participant-supervisor pairs by frequency and percent. 
Item responses were analyzed for each field test. A combined item analysis of responses 
of participant-supervisor pairs by frequency and percent was also provided. This data 
was presented in tables in Chapter IV. 

Research Question 1, “Is there a difference between the reactions of participants 
and the reactions of the participants’ supervisors to the programmatic effectiveness of the 
recently completed workforce training program?” was answered negatively. Item 
responses from both field tests for items 1 through 13, those items evaluating 
programmatic factors, indicated a consensus of response among the responding 
participants, consensus of that same response among the responding supervisors, and 
consensus of that same response among responding participant-supervisor pairs. The 
combined item responses of participant-supervisor pairs by frequency and percent for 
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programmatic factors indicated a consensus of response among a majority of responding 
pairs for 1 2 of the 1 3 items. Item 1 0, “I felt comfortable asking the instructor questions,” 
was the only item that had a larger percentage of no complete response rather than 
consensus of response. 

Information obtained by the researcher from the supervisors during the field tests 
indicated that some of those items evaluating programmatic factors were difficult for 
them to respond to since the supervisors were not participants in the training program. 
Therefore, several of the supervisors chose not to respond to some items, such as item 1 0. 
Some of the supervisors, however, had discussed the training program with their 
employees and had gathered enough information to answer many of the items. 

Research Question 2, “Is there a difference between the reactions of participants 
and the reactions of the participants’ supervisors to the personal effectiveness of the 
recently completed workforce training program?” was answered affirmatively, indicating 
a difference in response among the participants and their supervisors. Item responses 
from each field test indicated disagreement in response among a majority of the 
responding participants and a majority of the responding supervisors for four of the eight 
items evaluating personal factors. Three of those four items, items 18 through 20, 
indicated disagreement in response on both field tests. 

Item responses from Field Test I for item 1 7, “my desire to continue my education 
has increased” indicated a consensus of the same response among a majority of the 
participants, responding supervisors, and participant-supervisor pairs. In contrast, item 
responses from Field Test II for item 17 indicated a disagreement in response among 
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responding participants and supervisors. The difference in the results of the item 
responses for item 1 7 could be attributed to the fact that the training was voluntary in 
Field Test I and mandatory in Field Test II. 

The combined item responses of participant-supervisor pairs by frequency and 
percent for personal factors also indicated disagreement in response among a majority of 
the pairs for items 1 8 through 20. The combined item responses indicated a majority of 
no complete response on item 14 and item 1 7. Only two of the eight items evaluating 
personal factors had an overwhelmingly high percentage of agreement of response among 
responding participant-supervisor pairs. The percentage of agreement of response for 
item 1 6, “I am proud of my work and how it contributes to the success of the company,” 
was 73 percent. The percentage of agreement of response for item 21, “I am now able to 
perform my job better,” was 82 percent. 

Information obtained by the researcher from the supervisors during both field tests 
indicated their difficulty in responding to the items in the category of personal factors. 

The instrument was administered to the supervisors in Field Test I only two days after the 
conclusion of the training program. The instrument was administered to the supervisors 
in Field Test II only one week after the conclusion of the training program. It was the 
opinion of the supervisors that there was not enough time between the conclusion of the 
training program and the evaluation to observe measurable results. Findings from the 
literature (Shapiro, 1995; Gordon, 1996; Robinson, D. G., 1996; Parry, 1997; Rossi, 
Freeman, & Lipsey, 1 999) indicate that effective program evaluation is a continuous 
process, not one that occurs only at the conclusion of a training program. 
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Many of the items contained in the category of personal factors were specific to 
the participants’ feelings about their quality of life and academic achievement. This also 
made it very difficult for the supervisors to respond to those items without having ample 
time to observe the behavior of the participants or to discuss those items with them. 
Therefore, the supervisors chose either to disagree with many of the items or to not 
respond to them. Kirkpatrick (1998) states that personal factors involve a change in 
attitude, development or improvement of a skill, or a change in self-esteem. These 
factors can not be evaluated immediately at the conclusion of a training program. 

Research Question 3, “Is there a difference between the reactions of participants and the 
reactions of the participants’ supervisors to the external effectiveness of the recently 
completed workforce training program?” was answered affirmatively. Item responses 
from Field Test I showed a disagreement of response among a majority of the responding 
participants and a majority of the responding supervisors for three of the six items 
evaluating external factors. Item responses from Field Test II showed a disagreement of 
response among a majority of the responding participants and a majority of the 
responding supervisors for five of the six items evaluating external factors. Both field 
tests indicated disagreement of response among a majority of the responding participants 
and a majority of the responding supervisors for items 23, 26, and 27. Item 23 referred to 
an increase in the chances of making more money. Items 26 and 27 referred to the ability 
to get along better with co-workers and supervisors, respectively. 

Item responses from Field Test I for item 22, “my chances of getting a job, 
keeping my job, or advancing in my job have increased” indicated a consensus of the 
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same response among a majority of the participants, responding supervisors, and 
participant-supervisor pairs. In contrast, item responses from Field Test II for item 22 
indicated a disagreement in response among responding participants and supervisors. 

Item responses from Field Test I and Field Test II for item 24, “my ability to contribute to 
the well-being of my family has increased,” indicated results similar to those for item 22. 
The differences in the results of the item responses for items 22 and 24 could be a result 
of the training being voluntary in Field Test I and mandatory in Field Test II. 

Item 25, “I have been given or offered more responsibility in my job,” was the 
only item to receive a majority of a consensus of response among participants, 
supervisors, and participant-supervisor pairs for both field tests. The combined item 
responses of participant-supervisor pairs to all items evaluating external factors indicated 
a higher percentage of disagreement of response rather than agreement of response or no 
complete response. 

Information obtained by the researcher from the supervisors during both field tests 
indicated their difficulty in responding to the items in the category of external factors. 
This difficulty could again be attributed to the short period of time between the 
conclusion of the training program and the administration of the evaluation instrument. It 
was the opinion of the supervisors that there was not enough time between the conclusion 
of the training program and the evaluation to observe measurable results. Therefore, 
many of the supervisors chose either to disagree with some items or to not respond to 
them. External indicators of effectiveness are the most difficult to evaluate and require 
ample time to observe results (Kirkpatrick, 1998). 
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Conclusions 

Based on the data collected from the study the following conclusions were made: 

1 . The programmatic effectiveness of the workforce training program can be 
evaluated immediately at the conclusion of the training program using the 
instrument developed forthis study. 

2. The personal effectiveness of the workforce training program can not be 
evaluated immediately at the conclusion of the training program using the 
instrument developed for this study. 

3. The external effectiveness of the workforce training program can not be 
evaluated immediately at the conclusion of the training program using the 
instrument developed for this study. 

Recommendations 

Based on the data collected in this study, the following recommendations are 

made: 

1 . This study was conducted with two workforce training programs and a 
sample size of 34. It is recommended that the study be continued with 
several workforce training programs and a larger sample size. 

2. The evaluation instrument was administered to the supervisors in both 
field test groups within a short period of time following the conclusion of 
the workforce training program. It is recommended that the evaluation 




91 



81 



instrument be administered to the supervisors of the workforce training 
program participants after they have had ample time to discuss the training 
with the participants and observe changes in their behavior. 

3. Participants were able to respond to items evaluating programmatic factors 
immediately following the workforce training program. It is 
recommended that the participants evaluate the personal and external 
effectiveness of the workforce training program after they have had ample 
time to use the skills learned and recognize any changes in their behavior. 

4. Further research is needed in order to determine if there are differences in 
responses from participants in voluntary and mandatory workforce training 
programs. 
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LEGBSLATIVE ACCOUNTABSLITY REPORT ■ 

Please answer all questions as they provide the accountability and quality assurance for our system. Thbbiformation will remain CONFIDENTIAL and 
w ill be reported to the State Workforce Development Council and the State Legislature only as a consolidated report of a» businesses, industries, agencies end 
organizations served. Your original report will be returned to the below listed address. 



Company Atone; . 



Company-Address; 



_jCoUoge:_ 



(CM 



/Sfafo) 



_ Fiscal Y&an_ 
fZ ip Coda) 



PH. NO. 



Name 



Position 



1. information submitted by: 

n<uuc 

t please provide ihe following for only your ptent/business organization location: 

2a. Does your business employ fewer than 25 person*? P Yes Fewer than 100? PYes PNo 

2b. ‘ How many of your employees received training from a Career Center at your business/organization location? 

2c. * How many of your employees received training from a Career Center, at the Career Center, or away from your business? 

2d. Did your company or business take advantage of the Career Center's pre-employment training, employment certification program before 

hiring newemployees? □ Yes PNo ■ 

2e. Did your company take advantage of having all or part of your existing workforce assessed to determine a need or future need for training? 



Q Yes 



□ No 



3. "Estimate yourcompan/6 financial contribution to this training partnership. . 

\. As a result of training availability, did your company elect to expand operations P to locate additional plants P or to remain in 
Mississippi as a result of training? P NA □ 



5a. How much doyou expect the training received to Increase^ur annual sales or revenue? 

Q None O 50.000- 143,999 

□ $1,000-9.999 □ 150,000-499,999 

Q 10,000-49,999 □ 500,000 • 999,999 



□ Not applicable to our organization 
a 1,000,000-1,500.000 
Q 1,500,000* 

Q Non-Quantifiable 



□ 1,500,000+ 

Q Non<luantiflable 



5b. What annual savings do you anticipate from thetraWng received? P °“ r ^cm.ooo.oOO - 1,500.000 

O $1 000 - 9,999 □ 150,000-499,999 

□ 10,000-49,999 □ 500,000-999,999 

6. As a result of training, how many lobs were □ created □ retained? □ Not Applicable 

Create** □ 1 - 9 PM-74 fiefalrwf: * 



□ 1-9 

□ 10-24 

□ 25 -49 



P 50*74 

□ 75-149 

□ 150+ 



□ 1-9 

□ 10 -24 

□ 25 -49 



□ 50-74 

□ 75-149 

□ 1504 



7a. The average wage for Jobs created In question 6 Is: 

□ $ 5.25 • 6.49 

□ 6.50-7.99 

7b The average wage for Jobs retained In question 6 Is: 

□ $ 5.25 • 6.49 

□ 6.50-7.99 



P 8.00-9.99 
Q 10.00- 12-99 



□ 8.00-999 

□ 10.00 • 12.99 



□ 13.00-15.99 

□ 16.00 and above 



□ 13.00-15.99 

□ 16.00 and above 



□ .11 -.25 
Q .26 -.50 

□ Yes PNo 

□ Yes □ No 



□ .51 -.75 
0.76-1.50 



Q None n -H . •>< □ .51 - .75 o 1-50 and above 

□ $. 01 -.10 

8. Did the training provided meet your expectations? 

9. Would you use the college Career Centers again? 



□ TBD 



• 2b/2c - These are non-dupficated numbers, if a person waslralned wutifbe wiraSied In 2b and 250 In 2c. 

person receives foridlft training and safety traimng; that's 7reK You might be putting on a third shltt of 100 persons, but you assess 

site , count him/her only one time. 

-This includes both direct and in-kind costs such as salaries paWto employees receiving training, equipment purchased for training, ove me 

paid for training, contributions to employees' schooling, and paid time off to take the GED. et . Revised 3/15/99 
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External Factors: 

Financial: 

1 ) My chances of keeping my j ob have increased. 

2) My chances of making more money have increased. 

3) My chances of getting better opportunities for other jobs within my department or 
company have increased. 

4) My chances of getting a better job somewhere else have increased. 

5) My ability to contribute to the well-being of my family has increased. 

External Factors: 

Context-Based: 

6) I have been given or offered more responsibility on my job. 

7) I seem to get along better with my co-workers 

8) The positive interaction I have with my supervisor(s) has increased. 

Personal Factors: 

Quality of Life: 

9) I have more confidence in what I can do. 

1 0) My attitude about work has improved. 

11) lam proud of my work and how it contributes to the success of the company. 

12) My desire to continue my education has increased. 

Personal Factors: 

Academic Achievement: 

13) My ability to communicate with my co-workers has increased. 

14) My ability to communicate with my supervisor(s) has increased. 

15) My ability to solve problems by myself on my job has increased. 

16) I can do my job better. 

1 7) My ability to use my math skills on the job has increased. 

1 8) My ability to make decisions by myself on my job has increased. 

19) My ability to read job information like instructions and manuals has increased. 
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Programmatic Factors: 

Program Content: 

20) The information taught in the class helped me in my personal life. 

21) The information taught in the class helped me know more about my job. 

22) I am able to use the knowledge I have gained from this class away from my job. 

23) The information taught in the class relates directly to my current job 
responsibilities. 

24) The course material was organized. 

25) The course material was useful and easy to follow. 

26) I felt comfortable asking the teacher questions. 

27) The teacher gave me information about my progress. 

28) The teacher encouraged me. 

29) The teacher was available when I needed help. 

30) The teacher demonstrated thorough knowledge and understanding of the topic. 

31) The teacher presented the information in a clear, understandable, and professional 
manner. 

32) The teacher was well-prepared. 

Programmatic Factors: 

Program Structure: 

33) Getting information about the classes held at work was easy. 

34) The company encouraged me to attend classes. 

35) The company encouraged me to continue taking classes at work. 

36) The amount of time scheduled for this class was exactly what was needed to meet 
my objectives. 

37) My supervisor(s) encouraged me to attend classes at work. 

38) The teacher encouraged me to attend classes at work. 

39) The teacher knew how to teach the course material. 

40) The facilities for the class were comfortable. 

41) I would recommend this course to others. 
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WORKFORCE TRAINING PROGRAM EVALUATION 
Pilot Test Survey Questionnaire 
Directions: (To be read to the participant.) 

Please read each item below and respond by giving an item critique. Make any recommendations for changes to 
each item. Please note any areas of confusion which may cause the item to be misinterpreted by future participants 
or supervisors. Space is provided following each item for you to make your suggestions. 

Thank you for your assistance in this research study. 



I am completing this survey as a: 

workforce training program participant 

supervisor of a workforce training program participant. 

Programmatic Factors: 

Program Content: 

1) The course material was organized and easy to follow. 



2) The information taught in the class helped me (my employee) in my (his/her) personal life. 



3) The information taught in the class helped me (my employee) know more about my (his/her) job. 



4) The information taught in the class relates to my (my employee’s) current job responsibilities. 



Programmatic Factors: 

Program Structure and Facilities: 

5) Getting information about the classes held at work was easy. 



6 ) 



The amount of time scheduled for this class was exactly what was needed to meet my (my employee’s) 
objectives. 



7) The facilities for the class were comfortable. 



8) I would recommend this course to others. 



9) My (my employee’s) attendance at this training program was required. 



Programmatic Factors: 
Instructor: 



10) 


/ (my employee) felt comfortable asking the instructor questions. 


11) 


The instructor was encouraging. 


12) 


The instructor was available when / (my employee) needed help. 
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13) 


The instructor presented the information in a clear, understandable, and professional manner. 


Personal Factors: 
Quality of Life: 


14) 


I have (my employee has) more confidence in what I (he/she) can do. 


15) 


My (my employee’s) attitude about work has improved. 


16) 


I am (my employee is) proud of my (his/her) work and how it contributes to the success of the company. 


17) 


My (my employee’s) desire to continue my (his/her) education has increased. 



Personal Factors: 

Academic Achievement: 

18) My (my employee’s) ability to communicate with my (his/her) co-workers has increased. 



19) My (my employee’s) ability to communicate with my (his/her) supervisor has increased. 



20) My (my employee’s) ability to solve problems and make decisions (large or small) on my (his/her) job has 
increased. 



21) I (my employee) can do my (his/her) job better. 



External Factors: 

Financial: 

22) My (my employee’s) chances of getting a job, keeping my (his/her) job, or advancing on my (his/her) job 
have increased. 



23) My (my employee’s) chances of making more money have increased. 



24) My (my employee’s) ability to contribute to the well-being of my (his/her) family has increased. 



External Factors: 
Context-Based: 



25) 


I have (my employee has) been given or offered more responsibility on my (his/her) job. 


26) 


I get (my employee gets) along better with my (his/her) co-workers. 


27) 


I get (my employee gets) along better with my (his/her) supervisor. 
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PILOT TEST 



In-Person Introduction and Participant Consent Form (to be read to subjects) 

Hello, my name is Marjorie Taylor. I am a graduate student at Delta State University. I 
am conducting a research study on the effectiveness of workforce training programs in the 
Mississippi Delta. 

I have developed a survey questionnaire to assess the reaction of participants and their 
supervisors toward workforce training programs conducted by Coahoma Community College and 
Mississippi Delta Community College. As a recent participant (or a supervisor of a recent 
participant) in a workforce training program, you have been selected to participate in a pilot test 
of the survey questionnaire. 

I would like for you to read each item and respond by giving an item critique and make 
any recommendations for changes to the item. Please note any areas of confusion which may 
cause the item to be misinterpreted by future participants or supervisors. I have provided space 
following each item for you to make your suggestions. 

The survey questionnaire you complete today will not be marked in any way that will 
identify you. I will look at everyone’s information together. 

Do you understand what I have read to you? YES NO 

Do you voluntarily agree to take part in this pilot test? YES NO 



Signature of person giving consent Date 




Date 



Marjorie Taylor 
Delta State University 
Box C-l 

Cleveland, MS 38733 
(662) 846-4027 
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WORKFORCE TRAINING PROGRAM EVALUATION 
Field Test Survey Questionnaire 

Directions: (To be read to the participant. Participant items contain phrases in italics. Supervisor items 

contain phrases in parentheses.) 



Please read each item below and respond by placing a check in the appropriate column. Space is provided at the 
end of the questionnaire for any comments you may have related to the effectiveness of the workforce training 
program. 

Thank you for your assistance in this research study. 



I am completing this survey as a: 

workforce training program participant. Number 

supervisor of a workforce training program participant. Number 



ERIC 



ITEMS 


AGREE 


DISAGREE 


Programmatic Factors: 
Program Content: 






1) The course material was organized and easy to follow. 






2) The information taught in the class helped me (my employee) in my (his/her) 

personal life. 






3) The information taught in the class helped me (my employee) know more about 

my (his/her) current job. 






4) The information taught in the class relates to my (my employee's) current job 

responsibilities. 






Programmatic Factors: 

Program Structure and Facilities: 






5) Getting information about the classes offered at work was easy. 






6) The amount of time scheduled for this class was exactly what was needed to meet 

my (my employee’s) objectives. 






7) The facilities for the class were comfortable. 






8) I would recommend this course to others. 






9) My (my employee’s) attendance at this training program was mandatory. 






Programmatic Factors: 
Instructor: 






10) / (my employee) felt comfortable asking the instructor questions. 






1 1 ) The instructor encouraged my (my employee’s) participation in class. 






12) The instructor was available when I (my employee) needed help. 






13) The instructor presented the information in a clear, understandable, and 

professional manner. 






Personal Factors: 
Quality of Life: 






14) I have (my employee has) more confidence in performing my (his/her) job 

responsibilities. 
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1 5) My (my employee's) attitude about work has improved. 






1 6) I am (my employee is) proud of my (his/her) work and how it contributes to the 

success of the company. 






17) My (my employee’s) desire to continue my (his/her) education has increased. 






Personal Factors: 
Academic Achievement: 






18) My (my employee’s) ability to communicate with my (his/her) co-workers has 

improved. 






19) My (my employee’s) ability to communicate with my supervisor (me) has 

improved. 






20) My (my employee’s) ability to solve problems and make decisions, large or 

small, on my (his/her) job has increased. 






21) lam (my employee is) now able to perform my (his/her) job better. 






External Factors: 
Financial: 






22) My (my employee’s) chances of getting a job, keeping my (his/her) job, or 

advancing in my (his/her) job have increased. 






23) My (my employee’s) chances of making more money have increased. 






24) My (my employee’s) ability to contribute to the well-being of my (his/her) family 

has increased. 






External Factors: 
Context-Based: 






25) I have (my employee has) been given or offered more responsibility in my 

(his/her) job. 






26) I get (my employee gets) along better with my (his/her) co-workers. 






27) I get (my employee gets) along better with my supervisor (me). 







Comments: 



O 
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FIELD TEST 

In-Person Introduction and Participant Consent Form (to be read to subjects) 

Hello, my name is Maijorie Taylor. I am a graduate student at Delta State University. I 
am conducting a research study on the effectiveness of workforce training programs in the 
Mississippi Delta. 

I have developed a survey questionnaire to assess the reaction of participants and their 
supervisors to the effectiveness of workforce training programs conducted by Coahoma 
Community College and Mississippi Delta Community College. As a participant (or a supervisor 
of a participant) in a workforce training program, you have been selected to participate in a field 
test of the survey questionnaire. 

I am interested in the answers you give to questions about the workforce training program 
you (or your employee) participated in. I would like to know how you (or your employee) felt 
about the program content, the program structure and facilities, and the instructor. I would also 
like to know how the program affected your (or your employee’s) educational goals, work, and 
personal life. 

I will NOT keep your name with the questionnaire. No one will ever be told of the 
answers you give. I will look at everyone’s information together. 



Do you understand what I have read to you? 

Do you voluntarily agree to take part in this field test? 



YES 


NO 


YES 


NO 



Signature of person giving consent 



Date 



Signature of person getting consent 



Date 



Marjorie Taylor 
Delta State University 
Box €-1 

Cleveland, MS 38733 
(662) 846-4027 
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